Radiometric thermal imaging improvements for navigation systems and methods

ABSTRACT

Thermal imaging and navigation systems and related techniques are provided to improve the operation of manned or unmanned mobile platforms, including passenger vehicles. A system includes a thermal imaging device configured to be mounted on a vehicle. The thermal imaging device is configured to, when mounted on the vehicle, capture a first image of a scene encompassing a portion of the vehicle and capture a second image associated with a reflection of the scene from the portion of the vehicle. The system further includes a logic device configured to communicate with the thermal imaging device and determine a disparity map based on the first image and the second image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 63/167,644 filed on Mar. 29, 2021 and entitled“RADIOMETRIC THERMAL IMAGING IMPROVEMENTS FOR NAVIGATION SYSTEMS ANDMETHODS,” which is hereby incorporated by reference in its entirety.

This application is a continuation-in-part of International PatentApplication No. PCT/US2021/012554 filed Jan. 7, 2021 and entitled“VEHICULAR RADIOMETRIC CALIBRATION SYSTEMS AND METHODS,” which claimspriority to and the benefit of U.S. Provisional Patent Application No.62/959,602 filed on Jan. 10, 2020 and entitled “VEHICULAR RADIOMETRICCALIBRATION SYSTEMS AND METHODS,” which are hereby incorporated byreference in their entirety.

This application is related to U.S. patent application Ser. No.17/685,349 filed Mar. 2, 2022 and entitled “THERMAL IMAGING FORNAVIGATION SYSTEMS AND METHODS,” which claims priority to U.S.Provisional Patent Application No. 63/159,444 filed Mar. 10, 2021 andentitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” and U.S.Provisional Patent Application No. 63/167,632 filed Mar. 29, 2021 andentitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” which arehereby incorporated by reference in their entirety.

This application is related to U.S. patent application Ser. No.17/636,841 filed Feb. 18, 2022 which is a U.S. National Stage PatentApplication under 35 U.S.C. 371 of International Patent Application No.PCT/US2020/048450 filed Aug. 28, 2020 and entitled “MULTISPECTRALIMAGING FOR NAVIGATION SYSTEMS AND METHODS,” which claims priority toand the benefit of U.S. Provisional Patent Application No. 62/894,544filed on Aug. 30, 2019 and entitled “MULTISPECTRAL IMAGING FORNAVIGATION SYSTEMS AND METHODS,” which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The present invention relates generally to multispectral imaging and,more particularly, to systems and methods for multispectral imaging foruse in navigation of mobile platforms.

BACKGROUND

Modern mobile platforms, such as assisted or autopiloted manned andunmanned terrestrial vehicles and aircraft, including unmanned aerialvehicles (UAVs), remotely operated underwater vehicles (ROVs), unmanned(water) surface vehicles (USVs), and unmanned ground vehicles (UGVs),any of which may be configured as unmanned sensor platforms, are able tooperate over long distances and in all environments; rural, urban, andeven underwater. Operation of such systems may include real-timefeedback to a pilot and/or wireless transmissions between an unmannedplatform and a remote base station, which often includes a display toefficiently convey telemetry, imagery, and other sensor data captured bythe platform to an operator. An operator can often monitor auto orassisted navigation of, and if necessary pilot or otherwise control, amanned or unmanned mobile platform throughout an entire mission relyingsolely on imagery feedback or data received from the mobile platform.

Conventional imaging systems are often either too expensive and bulky orlack sufficient contrast under relatively common environmentalconditions to be used for reliable and safe auto or assisted navigationof a vehicle or other mobile platform. Thus, there is a need for compactimaging systems and related techniques to provide reliable sceneevaluation for use with navigation of mobile platforms.

SUMMARY

Multispectral navigation systems and related techniques are provided toimprove the operation of manned or unmanned mobile platforms, includingassisted-or-autopiloted manned vehicles and unmanned sensor or surveyplatforms. One or more embodiments of the described multispectralnavigation systems may advantageously include a multispectral imagingsystem including a multispectral imaging module, a communication moduleconfigured to establish a wireless communication link with a basestation associated with the mobile platform, an orientation and/orposition sensor configured to measure orientations and/or positions ofthe multispectral imaging system and/or a coupled mobile platform, acontroller to control operation of the communication module, theorientation and/or position sensor, and/or the mobile platform, and oneor more additional sensors to measure and provide sensor datacorresponding to maneuvering and/or other operation of the mobileplatform.

In various embodiments, such additional sensors may include a remotesensor system configured to capture sensor data of a survey area fromwhich a two and/or three-dimensional spatial map of the survey area maybe generated. For example, the navigation system may include one or morevisible spectrum, infrared, and/or ultraviolet cameras and/or otherremote sensor systems coupled to a mobile platform. The mobile platformmay generally be a flight platform (e.g., a manned aircraft, a UAS,and/or other flight platform), a terrestrial platform (e.g., a motorvehicle), or a water born platform (e.g., a watercraft or submarine).More generally, a multispectral imaging system for a multispectralnavigation system may be implemented as a multispectral autonomousvehicle imaging system (e.g., a MAVIS, for use with various autonomousor autopiloted mobile platforms or vehicles).

In one embodiment, a system includes a multispectral imaging systemincluding a multispectral imaging module configured to providemultispectral image data corresponding to a projected course for amobile platform, and a logic device configured to communicate with themultispectral imaging system. The logic device may be configured toreceive the multispectral image data corresponding to the projectedcourse; receive orientation and/or position data corresponding to themultispectral image data; and generate maneuvering obstacle informationcorresponding to the projected course based, at least in part, on acombination of the orientation and/or position data and themultispectral image data.

In another embodiment, a method includes receiving multispectral imagedata from a multispectral imaging system including a multispectralimaging module configured to provide multispectral image datacorresponding to a projected course for a mobile platform; receivingorientation and/or position data corresponding to the multispectralimage data; and generating maneuvering obstacle informationcorresponding to the projected course based, at least in part, on acombination of the orientation and/or position data and themultispectral image data.

In one embodiment, a system includes a thermal imaging system includinga thermal imaging module configured to provide thermal image datacorresponding to a projected course for a mobile platform, and a logicdevice configured to communicate with the thermal imaging system. Thelogic device may be configured to receive the thermal image datacorresponding to the projected course; and generate maneuvering obstacleinformation corresponding to the projected course based, at least inpart, on the thermal image data.

In another embodiment, a method includes receiving thermal image datafrom a thermal imaging system including a thermal imaging moduleconfigured to provide thermal image data corresponding to a projectedcourse for a mobile platform; and generating maneuvering obstacleinformation corresponding to the projected course based, at least inpart, on the thermal image data.

The scope of the invention is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the present invention will be affordedto those skilled in the art, as well as a realization of additionaladvantages thereof, by a consideration of the following detaileddescription of one or more embodiments. Reference will be made to theappended sheets of drawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of a multispectral navigation system inaccordance with an embodiment of the disclosure.

FIG. 2 illustrates a diagram of mobile platforms employing amultispectral navigation system in accordance with an embodiment of thedisclosure.

FIG. 3 illustrates a diagram of a multispectral imaging system for amultispectral navigation system in accordance with an embodiment of thedisclosure.

FIGS. 4-6 illustrate display views including imagery generated by amultispectral imaging system for a multispectral navigation system inaccordance with embodiments of the disclosure.

FIG. 7 illustrates a flow diagram of various operations to provideassisted or autopiloted navigation using a multispectral navigationsystem in accordance with embodiments of the disclosure.

FIGS. 8-10 illustrate display views including imagery generated by amultispectral imaging system for a multispectral navigation system inaccordance with embodiments of the disclosure.

FIG. 11 is a diagram illustrating the functional benefits associatedwith thermal imaging navigation systems in accordance with an embodimentof the disclosure.

FIG. 12 illustrates a diagram of a mobile platform employing a thermalimaging navigation system in accordance with an embodiment of thedisclosure.

FIG. 13A illustrates a data flow diagram of a mobile platform employinga thermal imaging navigation system in accordance with an embodiment ofthe disclosure.

FIG. 13B illustrates a block diagram of an update system for mobileplatforms employing thermal imaging navigation systems in accordancewith an embodiment of the disclosure.

FIGS. 14A-B illustrate display views including imagery generated by athermal imaging system for a thermal imaging navigation system inaccordance with embodiments of the disclosure.

FIG. 15 illustrates a flow diagram of various operations to provideassisted or autopiloted navigation including automated emergencybreaking using a thermal imaging navigation system in accordance withembodiments of the disclosure.

FIGS. 16A-F illustrate display views including imagery generated by athermal imaging system for a thermal imaging navigation system inaccordance with embodiments of the disclosure.

FIG. 17A illustrates a diagram of a model for atmospheric temperaturecompensation for a thermal imaging navigation system in accordance withembodiments of the disclosure.

FIG. 17B illustrates a plot of atmospheric transmission vs. range for athermal imaging navigation system in accordance with embodiments of thedisclosure.

FIG. 17C illustrates a plot of atmospheric transmission vs. wavelengthfor a thermal imaging navigation system in accordance with embodimentsof the disclosure.

FIG. 18 illustrates a diagram of ambient air and humidity sensors for athermal imaging navigation system in accordance with embodiments of thedisclosure.

FIG. 19 illustrates a display view including imagery generated by athermal imaging system for a thermal imaging navigation system inaccordance with embodiments of the disclosure.

FIG. 20 illustrates a diagram of a monocular range sensor for a rangedetermination using a thermal imaging navigation system in accordancewith embodiments of the disclosure.

FIG. 21 illustrates a diagram of a range determination using a thermalimaging navigation system in accordance with embodiments of thedisclosure.

FIG. 22 illustrates a flow diagram of various operations to generate arange map using a thermal imaging navigation system in accordance withembodiments of the disclosure.

FIGS. 23A-C illustrate display views including imagery generated by athermal imaging system for a thermal imaging navigation system inaccordance with embodiments of the disclosure.

Embodiments of the present invention and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures.

DETAILED DESCRIPTION

Multispectral navigation systems and related techniques are provided toimprove the operational flexibility and reliability of mobile platforms,including unmanned mobile sensor platforms. Imaging systems for use withan advanced driver assistance system (ADAS) typically acquire real-timevideo imagery of an immediate environment about a vehicle or mobileplatform. Such imagery can help a pilot (e.g., a human or autopilot)make decisions about mobile platform navigation, such as braking orevasive maneuvers. Contemporary commercial ADAS cameras produce imagesthat have limited or no spectral content: the imagery is either RGBcolor or monochrome visible-band. Such limited spectral content meansthat solid objects may have little or no contrast against the sky orother distant background, under many common environmental conditions(e.g., headings and/or times of day).

A real-world example might be where an ADAS fails to detect a relativelylarge featureless terrestrial obstruction (e.g., an enclosed trailer)relative to the sky/horizon. Such failure can be caused by limitedvisible differentiation (e.g., color, brightness) in the visiblespectrum. Such failure is not uncommon: a white object under an overcastsky will tend to have a brightness very similar to the sky due todiffuse reflection of the sky “dome.” Embodiments of a multispectralnavigation system described herein are much less likely to fail undersimilar conditions because it is extremely unlikely that a solid objectwill emit or reflect light similar to that in the sky over a combinationof spectral bands, which may include the visible spectrum and spectrumsoutside the visible, such as infrared and/or ultraviolet spectrums.

Scenes presented to an imaging navigation system often include areaswith very different near-infrared (NIR), visible (VIS), and longwave UV(LWUV) spectral content. A multispectral imaging system that issensitive to spectrums beyond the visible can more reliably determinethe composition of scene content, including being able to reliablydifferentiate sky from other vehicles, trees, shrubbery, structures, ora roadway, for example. A multispectral imaging system can thereforefeed an imaging navigation system with a much more nuanced data stream.

For example, simultaneous measurements in selected bands provides acoarse reflectance/emittance spectrum of surface materials in a scene.Materials like vegetation reflect sunlight with a distinctive spectralsignature, while the sky emits a different spectrum. Vehicles and roadsurfaces also appear different in such selected bands than they do inthe visible band. There are several advantages to using a small number(e.g., 2, 3, 4, or 5) of relatively wide spectral bands for amultispectral camera, as opposed to many closely-spaced relativelynarrow spectral channels. A wider spectral band (e.g., usuallyassociated with a wide bandpass filter) typically means that there willbe more scene flux reaching a detector and result in better exposures inlow light conditions. A wider spectral band also allows for shorterintegration times, which reduces motion blurring, a particular problemassociated with the edges of images captured by forward-looking systemsinstalled in fast-moving vehicles with relatively high angular rates ofmotion.

Such multispectral imaging capture and processing techniques can be usedwith aircraft as well, including unmanned aerial systems. For example, aUAV with autonomous operational capability may be implemented withimaging systems that can help a human or autopilot make decisions aboutwhat to do in different flight situations, including during takeoff,landing, and evasive action. A multispectral imaging system with imageanalytics can provide processed sensor information about the physicalenvironment, thereby helping the pilot steer away from obstacles, suchas a white blank billboard that might have the same visible-bandradiance as overcast sky behind the billboard. The multispectral imagerycan also help determine where vegetation is, which can help the mobileplatform avoid landing in a tree. The multispectral imagery can alsohelp the mobile platform know where the sky is in an imaged scenebecause the sky typically has a distinct multispectral signature. Themultispectral imagery can also help a UAS reliably and accurately locateother UASs in the sky, which can be extremely useful for aerialmaneuvers such as UAS swarming.

In addition to the above, embodiments may be made relatively compactly,thereby reducing size, weight, and power requirements (relative toconventional systems), and are therefore suitable for deployment invarious applications such as relatively small unmanned terrestrialvehicles and aircraft systems. Modern manned and unmanned mobileplatforms, including unmanned sensor platforms, such as unmanned aerialvehicles (UAVs), remotely operated underwater vehicles (ROVs), unmanned(water) surface vehicles (USVs), and unmanned ground vehicles (UGVs),are able to operate over long distances and in all environments. Suchsystems typically rely on a portable power source that can limit theirrange of travel. Embodiments described herein provide relativelylightweight, compact, and featureful multispectral navigation systemsthat typically increase the achievable range of such mobile platforms,including unmanned sensor platforms, which can be particularly helpfulwhen attempting to navigate within a survey area relatively quickly andexhaustively.

In various embodiments, multispectral imagery and/or other sensor datamay be transmitted to a base station, either in real-time or after asurvey, which may be configured to combine the sensor data with a map orfloor plan of a survey area to present the sensor data in a survey mapover the spatial extents of the map or floor plan. Such map or floorplan may be two or three dimensional. The survey map may be stored atthe base station and, if the base station includes a display, bepresented in real time as a graphically overlaid map to anoperator/user. During operation, such map may provide insight fornavigating a mobile platform or positioning a mobile platform forstationary observation, for example, or, if operation is to beundertaken in the same area at a future time, such map may provideinformation for route planning of future operations.

FIG. 1 illustrates a block diagram of multispectral navigation system100 in accordance with an embodiment of the disclosure. In someembodiments, system 100 may be configured to fly over a scene, through astructure, or approach a target and image or sense the scene, structure,or target, or portions thereof, using gimbal system 122 to aimmultispectral imaging system/sensor payload 140 and/or sensor cradle 128to aim environmental sensor 160 at the scene, structure, or target, orportions thereof. Resulting imagery and/or other sensor data may beprocessed (e.g., by sensor payload 140, platform 110, and/or basestation 130) and displayed to a user through use of user interface 132(e.g., one or more displays such as a multi-function display (MFD), aportable electronic device such as a tablet, laptop, or smart phone, orother appropriate interface) and/or stored in memory for later viewingand/or analysis. In some embodiments, system 100 may be configured touse such imagery and/or other sensor data to control operation ofplatform 110, sensor payload 140, and/or environmental sensor 160, asdescribed herein, such as controlling gimbal system 122 to aim sensorpayload 140 towards a particular direction or controlling propulsionsystem 124 to move and/or orient platform 110 to a desiredposition/orientation in a scene or structure or relative to a target.

In additional embodiments, system 100 may be configured to use platform110 and/or sensor cradle 128 to position and/or orient environmentalsensor 160 at or relative to the scene, structure, or target, orportions thereof. Resulting sensor data may be processed (e.g., byenvironmental sensor 160, platform 110, and/or base station 130) anddisplayed to a user through use of user interface 132 (e.g., one or moredisplays such as a multi-function display (MFD), a portable electronicdevice such as a tablet, laptop, or smart phone, or other appropriateinterface) and/or stored in memory for later viewing and/or analysis. Insome embodiments, system 100 may be configured to use such sensor datato control operation of platform 110 and/or environmental sensor 160, asdescribed herein, such as controlling propulsion system 124 to moveand/or orient platform 110 to a desired position in a scene or structureor relative to a target.

In the embodiment shown in FIG. 1, multispectral navigation system 100includes platform 110, optional base station 130, and at least onemultispectral imaging system 140. Platform 110 may be a mobile platformconfigured to move or fly and position multispectral imaging system 140and/or environmental sensor 160 (e.g., relative to a designated ordetected target). As shown in FIG. 1, platform 110 may include one ormore of a controller 112, an orientation sensor 114, agyroscope/accelerometer 116, a global navigation satellite system (GNSS)118, a communication module 120, a gimbal system 122, a propulsionsystem 124, a sensor cradle 128, and other modules 126. Operation ofplatform 110 may be substantially autonomous and/or partially orcompletely controlled by optional base station 130, which may includeone or more of a user interface 132, a communication module 134, andother modules 136. In other embodiments, platform 110 may include one ormore of the elements of base station 130, such as with various types ofmanned aircraft, terrestrial vehicles, and/or surface or subsurfacewatercraft.

Sensor payload 140 and/or environmental sensor 160 may be physicallycoupled to platform 110 and be configured to capture sensor data (e.g.,visible spectrum images, infrared images, ultraviolet images, narrowaperture radar data, analyte sensor data, directional radiation data,and/or other sensor data) of a target position, area, and/or object(s)as selected and/or framed by operation of platform 110 and/or basestation 130. In some embodiments, one or more of the elements of system100 may be implemented in a combined housing or structure that can becoupled to or within platform 110 and/or held or carried by a user ofsystem 100.

Controller 112 may be implemented as any appropriate logic device (e.g.,processing device, microcontroller, processor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), memorystorage device, memory reader, or other device or combinations ofdevices) that may be adapted to execute, store, and/or receiveappropriate instructions, such as software instructions implementing acontrol loop for controlling various operations of platform 110 and/orother elements of system 100, for example. Such software instructionsmay also implement methods for processing infrared images and/or othersensor signals, determining sensor information, providing user feedback(e.g., through user interface 132), querying devices for operationalparameters, selecting operational parameters for devices, or performingany of the various operations described herein (e.g., operationsperformed by logic devices of various devices of system 100).

In addition, a non-transitory medium may be provided for storing machinereadable instructions for loading into and execution by controller 112.In these and other embodiments, controller 112 may be implemented withother components where appropriate, such as volatile memory,non-volatile memory, one or more interfaces, and/or various analogand/or digital components for interfacing with devices of system 100.For example, controller 112 may be adapted to store sensor signals,sensor information, parameters for coordinate frame transformations,calibration parameters, sets of calibration points, and/or otheroperational parameters, over time, for example, and provide such storeddata to a user using user interface 132. In some embodiments, controller112 may be integrated with one or more other elements of platform 110,for example, or distributed as multiple logic devices within platform110, base station 130, and/or sensor payload 140.

In some embodiments, controller 112 may be configured to substantiallycontinuously monitor and/or store the status of and/or sensor dataprovided by one or more elements of platform 110, sensor payload 140,environmental sensor 160, and/or base station 130, such as the positionand/or orientation of platform 110, sensor payload 140, and/or basestation 130, for example, and the status of a communication linkestablished between platform 110, sensor payload 140, environmentalsensor 160, and/or base station 130. Such communication links may beconfigured to be established and then transmit data between elements ofsystem 100 substantially continuously throughout operation of system100, where such data includes various types of sensor data, controlparameters, and/or other data.

Orientation sensor 114 may be implemented as one or more of a compass,float, accelerometer, and/or other device capable of measuring anorientation of platform 110 (e.g., magnitude and direction of roll,pitch, and/or yaw, relative to one or more reference orientations suchas gravity and/or Magnetic North), gimbal system 122, imagingsystem/sensor payload 140, and/or other elements of system 100, andproviding such measurements as sensor signals and/or data that may becommunicated to various devices of system 100. Gyroscope/accelerometer116 may be implemented as one or more electronic sextants, semiconductordevices, integrated chips, accelerometer sensors, accelerometer sensorsystems, or other devices capable of measuring angularvelocities/accelerations and/or linear accelerations (e.g., directionand magnitude) of platform 110 and/or other elements of system 100 andproviding such measurements as sensor signals and/or data that may becommunicated to other devices of system 100 (e.g., user interface 132,controller 112).

GNSS 118 may be implemented according to any global navigation satellitesystem, including a GPS, GLONASS, and/or Galileo based receiver and/orother device capable of determining absolute and/or relative position ofplatform 110 (e.g., or an element of platform 110) based on wirelesssignals received from space-born and/or terrestrial sources (e.g.,eLoran, and/or other at least partially terrestrial systems), forexample, and capable of providing such measurements as sensor signalsand/or data (e.g., coordinates) that may be communicated to variousdevices of system 100. In some embodiments, GNSS 118 may include analtimeter, for example, or may be used to provide an absolute altitude.

Communication module 120 may be implemented as any wired and/or wirelesscommunication module configured to transmit and receive analog and/ordigital signals between elements of system 100. For example,communication module 120 may be configured to receive flight controlsignals and/or data from base station 130 and provide them to controller112 and/or propulsion system 124. In other embodiments, communicationmodule 120 may be configured to receive images and/or other sensorinformation (e.g., visible spectrum, infrared, and/or ultraviolet stillimages or video images) from sensor payload 140 and relay the sensordata to controller 112 and/or base station 130. In further embodiments,communication module 120 may be configured to receive sensor data and/orother sensor information from environmental sensor 160 and relay thesensor data to controller 112 and/or base station 130. In someembodiments, communication module 120 may be configured to supportspread spectrum transmissions, for example, and/or multiple simultaneouscommunications channels between elements of system 100. Wirelesscommunication links may include one or more analog and/or digital radiocommunication links, such as WiFi and others, as described herein, andmay be direct communication links established between elements of system100, for example, or may be relayed through one or more wireless relaystations configured to receive and retransmit wireless communications.

In some embodiments, communication module 120 may be configured tomonitor the status of a communication link established between platform110, sensor payload 140, and/or base station 130. Such statusinformation may be provided to controller 112, for example, ortransmitted to other elements of system 100 for monitoring, storage, orfurther processing, as described herein. Communication links establishedby communication module 120 may be configured to transmit data betweenelements of system 100 substantially continuously throughout operationof system 100, where such data includes various types of sensor data,control parameters, and/or other data, as described herein.

In some embodiments, gimbal system 122 may be implemented as an actuatedgimbal mount, for example, that may be controlled by controller 112 tostabilize sensor payload 140 relative to a target or to aim sensorpayload 140 according to a desired direction and/or relative position.As such, gimbal system 122 may be configured to provide a relativeorientation of sensor payload 140 (e.g., relative to an orientation ofplatform 110) to controller 112 and/or communication module 120 (e.g.,gimbal system 122 may include its own orientation sensor 114). In otherembodiments, gimbal system 122 may be implemented as a gravity drivenmount (e.g., non-actuated). In various embodiments, gimbal system 122may be configured to provide power, support wired communications, and/orotherwise facilitate operation of articulated sensor/sensor payload 140.In further embodiments, gimbal system 122 may be configured to couple toa laser pointer, range finder, and/or other device, for example, tosupport, stabilize, power, and/or aim multiple devices (e.g., sensorpayload 140 and one or more other devices) substantially simultaneously.In alternative embodiments, multispectral imaging system/sensor payload140 may be fixed to mobile platform 110 such that gimbal system 122 isimplemented as a fixed perspective mounting system for sensor payload140.

Propulsion system 124 may be implemented as one or more propellers,turbines, or other thrust-based propulsion systems, and/or other typesof propulsion systems that can be used to provide motive force and/orlift to platform 110 and/or to steer platform 110. In some embodiments,propulsion system 124 may include multiple propellers (e.g., a tri,quad, hex, oct, or other type “copter”) that can be controlled (e.g., bycontroller 112) to provide lift and motion for platform 110 and toprovide an orientation for platform 110. In other embodiments,propulsion system 124 may be configured primarily to provide thrustwhile other structures of platform 110 provide lift, such as in a fixedwing embodiment (e.g., where wings provide the lift) and/or an aerostatembodiment (e.g., balloons, airships, hybrid aerostats).

In various embodiments, propulsion system 124 may be implemented with aportable power supply, such as a battery and/or a combustionengine/generator and fuel supply, for example, that may be coupled to atransmission and/or drive train for propulsion system 124 and/orplatform 110. In further embodiments, propulsion system 124 may beimplemented with braking system 125, for example, which may be used orcontrolled to damp or eliminate motion of platform 110, such as anelectromechanically controlled disk or drum based braking system for usewith terrestrial vehicles, including passenger vehicles.

Other modules 126 may include other and/or additional sensors,actuators, communication modules/nodes, and/or user interfaces/interfacedevices, for example, and may be used to provide additionalenvironmental information related to operation of platform 110, forexample. In some embodiments, other modules 126 may include a humiditysensor, a wind and/or water temperature sensor, a barometer, analtimeter, an analyte detection system, a radar system, a proximitysensor, a visible spectrum camera or infrared/thermal camera (with anadditional mount), an irradiance detector, and/or other environmentalsensors providing measurements and/or other sensor signals that can bedisplayed to a user and/or used by other devices of system 100 (e.g.,controller 112) to provide operational control of platform 110 and/orsystem 100.

In some embodiments, other modules 126 may include one or more actuatedand/or articulated devices (e.g., multi-spectrum active illuminators,visible and/or IR cameras, radars, sonars, and/or other actuateddevices) coupled to platform 110, where each actuated device includesone or more actuators adapted to adjust an orientation of the device,relative to platform 110, in response to one or more control signals(e.g., provided by controller 112). In particular, other modules 126 mayinclude a stereo vision system configured to provide image data that maybe used to calculate or estimate a position of platform 110, forexample, or to calculate or estimate a relative position of anavigational hazard in proximity to platform 110. In variousembodiments, controller 130 may be configured to use such proximityand/or position information to help safely pilot platform 110 and/ormonitor communication link quality, as described herein.

Ranging sensor system 127 may be implemented as a radar, sonar, lidar,and/or other ranging sensor system fixed relative to platform 110,imaging system 140, and/or environmental sensor 160, and be configuredto provide two and/or three dimensional ranging sensor datacorresponding to a depth map overlapping a field of view of imagingsystem 140 and/or environmental sensor 160 and/or substantially centeredabout the optical axis of imaging system 140 and/or environmental sensor160.

In embodiments where ranging sensor system 127 is implemented as a radarsystem, ranging sensor system 127 may be implemented as one or moreelectrically and/or mechanically coupled controllers, transmitters,receivers, transceivers, signal processing logic devices, variouselectrical components, antenna elements of various shapes and sizes,multichannel antennas/antenna modules, radar assemblies, assemblybrackets, and/or various actuators adapted to adjust orientations of anyof the components ranging sensor system 127, as described herein. Forexample, in various embodiments, ranging sensor system 127 may beimplemented according to various radar system arrangements that can beused to detect features of and objects on or above a terrestrial surfaceor a surface of a body of water, for instance, and/or their relativevelocities (e.g., their Doppler velocities).

More generally, ranging sensor system 127 may be configured to emit one,multiple, or a series of radar beams (e.g., remote sensor beams),receive corresponding radar returns (e.g., remote sensor returns), andconvert the radar returns into radar data and/or imagery (e.g., remotesensor image data), such as one or more intensity plots and/oraggregation of intensity plots indicating a relative position,orientation, and/or other characteristics of structures, weatherphenomena, waves, other mobile structures, surface boundaries, and/orother maneuvering obstacles and/or objects reflecting the radar beamsback at ranging sensor system 127. Ranging sensor system 127 may beconfigured to provide such data and/or imagery to a user interface ofplatform 110 and/or base station 130 for display to a user, for example,or to controller 112 for additional processing, as described herein.Moreover, such data may be used to generate one or more chartscorresponding to AIS data, ARPA data, MARPA data, and or one or moreother target tracking and/or identification protocols.

In some embodiments, ranging sensor system 127 may be implemented usinga compact design, where multiple radar antennas, sensors, and/orassociated processing devices are located within a single radar assemblyhousing that is configured to interface with the rest of system 100through a single cable providing both power and communications to andfrom ranging sensor system 127. In some embodiments, ranging sensorsystem 127 may include orientation and/or position sensors configured tohelp provide two or three-dimensional waypoints, increase radar dataand/or imagery quality, and/or provide highly accurate radar image data,as described herein.

Conventional radar systems can be expensive and bulky and typicallycannot be used to provide relatively accurate and/or distortion freeradar image data. Embodiments of ranging sensor system 127 include lowcost single, dual, and/or multichannel (e.g., synthetic aperture) radarsystems that can be configured to produce detailed two andthree-dimensional radar data and/or imagery. In some embodiments,ranging sensor system 127 may consolidate electronics and transducersinto a single waterproof package to reduce size and cost, for example,and may be implemented with a single connection to other devices ofsystem 100 (e.g., via an Ethernet cable with power over Ethernet, anintegral power cable, and/or other communication and/or powertransmission conduits integrated into a single interface cable).

In various embodiments, ranging sensor system 127 may be implementedwith its own dedicated orientation and/or position sensors (e.g.,similar to orientation sensor 114, gyroscope/accelerometer 116, and/orGNSS 118) that may be incorporated within a radar assembly housing toprovide three dimensional orientations and/or positions of the radarassembly and/or antenna(s) for use when processing or post processingradar data for display. The sensor information can be used to correctfor movement of the radar assembly during and/or between beam emissionsto provide improved alignment of corresponding radar returns/samples,for example, and/or to generate imagery based on the measuredorientations and/or positions of the radar assembly/antenna. In otherembodiments, an external orientation and/or position sensor can be usedalone or in combination with an integrated sensor or sensors.

In embodiments where ranging sensor system 127 is implemented with oneor more position sensors, ranging sensor system 127 may be configured toprovide a variety of radar data and/or imagery enhancements. Forexample, ranging sensor system 127 may be configured to provide accuratepositioning and/or orienting of radar data and/or user-defined waypointsremote from platform 110. Similarly, ranging sensor system 127 may beconfigured to provide accurate two and/or three-dimensional aggregationand/or display of a series of radar data; without either orientationdata or position data to help determine a track or heading, a radarsystem typically assumes a straight track, which can cause imageartifacts and/or other inaccuracies in corresponding radar data and/orimagery. Additionally, when implemented with a position sensor, rangingsensor system 127 may be configured to generate accurate and detailedintensity plots of maneuvering obstacles without access to amagnetometer.

In embodiments where ranging sensor system 127 is implemented with anorientation and/or position sensor, ranging sensor system 127 may beconfigured to store such location/position information along with othersensor information (radar returns, temperature measurements, textdescriptions, altitude, platform speed, and/or other sensor and/orcontrol information) available to system 100. In some embodiments,controller 112 may be configured to generate a look up table so that auser can select desired configurations of ranging sensor system 127 fora particular location or to coordinate with some other sensorinformation. Alternatively, an automated adjustment algorithm can beused to select optimum configurations based on the sensor information.

In various embodiments, ranging sensor system 127 may also beimplemented with a colocated imaging system (e.g., imaging system 140),which may include one or more various types of imaging modules that maybe incorporated within the radar assembly housing to provide image datasubstantially contemporaneous with radar data for use when processing orpost processing radar data for display. The image data can be used toimprove operator understanding of the radar data and to increase theoverall functionality of system 100. For example, embodiments mayinclude one or multiple imaging modules such that the imaging modulesrotate with a radar antenna of ranging sensor system 127 to generate apanorama corresponding to a radar plan position indicator (PPI) displayview. Embodiments provide methods of data processing, data fusion, anddisplaying the data and user interaction, as described herein. Infurther embodiments, ranging sensor system 127 may be implemented as aradar system implemented as and/or configured to operate similar toembodiments described in U.S. patent application Ser. No. 16/0007,908filed Jun. 13, 2018 and entitled “VEHICLE BASED RADAR UPSAMPLING,” nowU.S. Pat. No. 10,928,512, which are hereby incorporated by reference intheir entirety.

In various embodiments, sensor cradle 128 may be implemented as alatching mechanism that may be permanently mounted to platform 110 toprovide a mounting position and/or orientation for environmental sensor160 relative to a center of gravity of platform 110, relative topropulsion system 124, and/or relative to other elements of platform110. In addition, sensor cradle 128 may be configured to provide power,support wired communications, and/or otherwise facilitate operation ofenvironmental sensor 160, as described herein. As such, sensor cradle128 may be configured to provide a power, telemetry, and/or other sensordata interface between platform 110 and environmental sensor 160. Insome embodiments, gimbal system 122 may be implemented similarly tosensor cradle 128, and vice versa.

For example, sensor cradle 128 may be implemented as an actuated gimbalmount, for example, that may be controlled by controller 112 tostabilize environmental sensor 160 relative to a target or to aimenvironmental sensor 160 according to a desired direction and/orrelative position. As such, sensor cradle 128 may be configured toprovide a relative orientation of environmental sensor 160 (e.g.,relative to an orientation of platform 110) to controller 112 and/orcommunication module 120 (e.g., sensor cradle 128 may include its ownorientation sensor 114). In other embodiments, sensor cradle 128 may beimplemented as a gravity driven mount (e.g., non-actuated). In furtherembodiments, sensor cradle 128 may be configured to couple to a laserpointer, range finder, and/or other device, for example, to support,stabilize, power, and/or aim multiple devices (e.g., environmentalsensor 160 and one or more other devices) substantially simultaneously.

User interface 132 of base station 130 may be implemented as one or moreof a display, a touch screen, a keyboard, a mouse, a joystick, a knob, asteering wheel, a yoke, and/or any other device capable of acceptinguser input and/or providing feedback to a user. In various embodiments,user interface 132 may be adapted to provide user input (e.g., as a typeof signal and/or sensor information transmitted by communication module134 of base station 130) to other devices of system 100, such ascontroller 112. User interface 132 may also be implemented with one ormore logic devices (e.g., similar to controller 112) that may be adaptedto store and/or execute instructions, such as software instructions,implementing any of the various processes and/or methods describedherein. For example, user interface 132 may be adapted to formcommunication links, transmit and/or receive communications (e.g.,infrared images and/or other sensor signals, control signals, sensorinformation, user input, and/or other information), for example, or toperform various other processes and/or methods described herein.

In one embodiment, user interface 132 may be adapted to display a timeseries of various sensor information and/or other parameters as part ofor overlaid on a graph or map, which may be referenced to a positionand/or orientation of platform 110 and/or other elements of system 100.For example, user interface 132 may be adapted to display a time seriesof positions, headings, and/or orientations of platform 110 and/or otherelements of system 100 overlaid on a geographical map, which may includeone or more graphs indicating a corresponding time series of actuatorcontrol signals, sensor information, and/or other sensor and/or controlsignals.

In some embodiments, user interface 132 may be adapted to accept userinput including a user-defined target destination, heading, waypoint,route, and/or orientation for an element of system 100, for example, andto generate control signals to cause platform 110 to move according tothe target destination, heading, route, and/or orientation, or to aimsensor payload 140 or environmental sensor 160 accordingly. In otherembodiments, user interface 132 may be adapted to accept user inputmodifying a control loop parameter of controller 112, for example.

In further embodiments, user interface 132 may be adapted to accept userinput including a user-defined target attitude, orientation, and/orposition for an actuated or articulated device (e.g., sensor payload 140or environmental sensor 160) associated with platform 110, for example,and to generate control signals for adjusting an orientation and/orposition of the actuated device according to the target attitude,orientation, and/or position. Such control signals may be transmitted tocontroller 112 (e.g., using communication modules 134 and 120), whichmay then control platform 110 accordingly.

Communication module 134 may be implemented as any wired and/or wirelesscommunication module configured to transmit and receive analog and/ordigital signals between elements of system 100. For example,communication module 134 may be configured to transmit flight controlsignals from user interface 132 to communication module 120 or 144. Inother embodiments, communication module 134 may be configured to receivesensor data (e.g., visible spectrum, infrared, and/or ultraviolet stillimages or video images, or other sensor data) from sensor payload 140.In some embodiments, communication module 134 may be configured tosupport spread spectrum transmissions, for example, and/or multiplesimultaneous communications channels between elements of system 100. Invarious embodiments, communication module 134 may be configured tomonitor the status of a communication link established between basestation 130, sensor payload 140, and/or platform 110 (e.g., includingpacket loss of transmitted and received data between elements of system100, such as with digital communication links), as described herein.Such status information may be provided to user interface 132, forexample, or transmitted to other elements of system 100 for monitoring,storage, or further processing, as described herein.

Other modules 136 of base station 130 may include other and/oradditional sensors, actuators, communication modules/nodes, and/or userinterface devices used to provide additional environmental informationassociated with base station 130, for example. In some embodiments,other modules 136 may include a humidity sensor, a wind and/or watertemperature sensor, a barometer, a radar system, a visible spectrumcamera, an infrared or thermal camera, a GNSS, and/or otherenvironmental sensors providing measurements and/or other sensor signalsthat can be displayed to a user and/or used by other devices of system100 (e.g., controller 112) to provide operational control of platform110 and/or system 100 or to process sensor data to compensate forenvironmental conditions, such as an water content in the atmosphereapproximately at the same altitude and/or within the same area asplatform 110 and/or base station 130, for example. In some embodiments,other modules 136 may include one or more actuated and/or articulateddevices (e.g., multi-spectrum active illuminators, visible and/or IRcameras, radars, sonars, and/or other actuated devices), where eachactuated device includes one or more actuators adapted to adjust anorientation of the device in response to one or more control signals(e.g., provided by user interface 132).

In embodiments where imaging system/sensor payload 140 is implemented asan imaging device, imaging system/sensor payload 140 may include imagingmodule 142, which may be implemented as a cooled and/or uncooled arrayof detector elements, such as visible spectrum, infrared, and/orultraviolet sensitive detector elements, including quantum well infraredphotodetector elements, bolometer or microbolometer based detectorelements, type II superlattice based detector elements, and/or otherinfrared spectrum detector elements that can be arranged in a focalplane array (FPA) (e.g., along with other detector elements sensitive toother spectrums). In various embodiments, imaging module 142 may beimplemented with a complementary metal oxide semiconductor (CMOS) basedFPA of detector elements that are sensitive to portions of the visible,near-infrared (NIR), and long wave ultraviolet (LWUV) spectrumssimultaneously. In various embodiments, imaging module 142 may includeone or more logic devices (e.g., similar to controller 112) that can beconfigured to process imagery captured by detector elements of imagingmodule 142 before providing the imagery to memory 146 or communicationmodule 144. More generally, imaging module 142 may be configured toperform any of the operations or methods described herein, at least inpart, or in combination with controller 112 and/or user interface 132.

In some embodiments, sensor payload 140 may be implemented with a secondor additional imaging modules similar to imaging module 142, forexample, that may include detector elements configured to detect otherelectromagnetic spectrums, such as visible light, thermal, ultraviolet,and/or other electromagnetic spectrums or subsets of such spectrums. Invarious embodiments, such additional imaging modules may be calibratedor registered to imaging module 142 such that images captured by eachimaging module occupy a known and at least partially overlapping fieldof view of the other imaging modules, thereby allowing differentspectrum images to be geometrically registered to each other (e.g., byscaling and/or positioning). In some embodiments, different spectrumimages may be registered to each other using pattern recognitionprocessing in addition or as an alternative to reliance on a knownoverlapping field of view.

Communication module 144 of sensor payload 140 may be implemented as anywired and/or wireless communication module configured to transmit andreceive analog and/or digital signals between elements of system 100.For example, communication module 144 may be configured to transmitimages from imaging module 142 to communication module 120 or 134. Inother embodiments, communication module 144 may be configured to receivecontrol signals (e.g., control signals directing capture, focus,selective filtering, and/or other operation of sensor payload 140) fromcontroller 112 and/or user interface 132. In some embodiments,communication module 144 may be configured to support spread spectrumtransmissions, for example, and/or multiple simultaneous communicationschannels between elements of system 100. In various embodiments,communication module 144 may be configured to monitor the status of acommunication link established between sensor payload 140, base station130, and/or platform 110 (e.g., including packet loss of transmitted andreceived data between elements of system 100, such as with digitalcommunication links), as described herein. Such status information maybe provided to imaging module 142, for example, or transmitted to otherelements of system 100 for monitoring, storage, or further processing,as described herein.

Memory 146 may be implemented as one or more machine readable mediumsand/or logic devices configured to store software instructions, sensorsignals, control signals, operational parameters, calibrationparameters, infrared images, and/or other data facilitating operation ofsystem 100, for example, and provide it to various elements of system100. Memory 146 may also be implemented, at least in part, as removablememory, such as a secure digital memory card for example including aninterface for such memory.

Orientation sensor 148 of sensor payload 140 may be implemented similarto orientation sensor 114 or gyroscope/accelerometer 116, and/or anyother device capable of measuring an orientation of sensor payload 140,imaging module 142, and/or other elements of sensor payload 140 (e.g.,magnitude and direction of roll, pitch, and/or yaw, relative to one ormore reference orientations such as gravity and/or Magnetic North) andproviding such measurements as sensor signals that may be communicatedto various devices of system 100. Gyroscope/accelerometer (e.g., angularmotion sensor) 150 of sensor payload 140 may be implemented as one ormore electronic sextants, semiconductor devices, integrated chips,accelerometer sensors, accelerometer sensor systems, or other devicescapable of measuring angular velocities/accelerations (e.g., angularmotion) and/or linear accelerations (e.g., direction and magnitude) ofsensor payload 140 and/or various elements of sensor payload 140 andproviding such measurements as sensor signals that may be communicatedto various devices of system 100. GNSS 149 may be implemented similar toGNSS 118 and/or any other device capable of measuring a position ofsensor payload 140, imaging module 142, and/or other elements of sensorpayload 140 and providing such measurements as sensor signals that maybe communicated to various devices of system 100.

Other modules 152 of sensor payload 140 may include other and/oradditional sensors, actuators, communication modules/nodes, cooled oruncooled optical filters, and/or user interface devices used to provideadditional environmental information associated with sensor payload 140,for example. In some embodiments, other modules 152 may include ahumidity sensor, a wind and/or water temperature sensor, a barometer, aradar system, a visible spectrum camera, an infrared camera, a GNSS,and/or other environmental sensors providing measurements and/or othersensor signals that can be displayed to a user and/or used by imagingmodule 142 or other devices of system 100 (e.g., controller 112) toprovide operational control of platform 110 and/or system 100 or toprocess imagery to compensate for environmental conditions.

In various embodiments, environmental sensor/sensor payload 160 may beimplemented as an environmental sensor configured to generateenvironmental sensor data corresponding to the environment surroundingplatform 110. In the embodiment shown in FIG. 1, environmental sensor160 includes sensor controller 162, memory 163, communication module164, sensor assembly 166, orientation and/or position sensor (OPS) 167,power supply 168, and other modules 170. In various embodiments, sensorassembly 166 may be implemented with sensor elements configured todetect the presence of and/or generate sensor data corresponding tohazardous analytes, ionizing radiation, emissivities, thermal radiation,radio frequency signals, and/or other environmental conditions proximateto or in view of platform 110 and/or environmental sensor 160.

Sensor controller 162 may be implemented as one or more of anyappropriate logic device (e.g., processing device, microcontroller,processor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), memory storage device, memory reader, orother device or combinations of devices) that may be adapted to execute,store, and/or receive appropriate instructions, such as softwareinstructions implementing a control loop for controlling variousoperations of environmental sensor 160 and/or other elements ofenvironmental sensor 160, for example. Such software instructions mayalso implement methods for processing sensor signals, determining sensorinformation, providing user feedback (e.g., through user interface 132),querying devices for operational parameters, selecting operationalparameters for devices, or performing any of the various operationsdescribed herein.

In addition, a non-transitory medium may be provided for storing machinereadable instructions for loading into and execution by sensorcontroller 162. In these and other embodiments, sensor controller 162may be implemented with other components where appropriate, such asvolatile memory, non-volatile memory, one or more interfaces, and/orvarious analog and/or digital components for interfacing with modules ofenvironmental sensor 160 and/or devices of system 100. For example,sensor controller 162 may be adapted to store sensor signals, sensorinformation, parameters for coordinate frame transformations,calibration parameters, sets of calibration points, and/or otheroperational parameters, over time, for example, and provide such storeddata to a user using user interface 132. In some embodiments, sensorcontroller 162 may be integrated with one or more other elements ofenvironmental sensor 160, for example, or distributed as multiple logicdevices within platform 110, base station 130, and/or environmentalsensor 160.

In some embodiments, sensor controller 162 may be configured tosubstantially continuously monitor and/or store the status of and/orstore sensor data provided by one or more elements of sensor assembly166 of environmental sensor 160, such as the position and/or orientationof platform 110, environmental sensor 160, and/or base station 130, forexample, and the status of a communication link established betweenplatform 110, environmental sensor 160, and/or base station 130. Suchcommunication links may be configured to be established and thentransmit data between elements of system 100 substantially continuouslythroughout operation of system 100, where such data includes varioustypes of sensor data, control parameters, and/or other data.

Memory 163 may be implemented as one or more machine readable mediumsand/or logic devices configured to store software instructions, sensorsignals, control signals, operational parameters, calibrationparameters, sensor data, and/or other data facilitating operation ofenvironmental sensor 160 and/or other elements of system 100, forexample, and provide it to various elements of system 100. Memory 163may also be implemented, at least in part, as removable memory, such asa secure digital memory card for example including an interface for suchmemory.

Communication module 164 of environmental sensor 160 may be implementedas any wired and/or wireless communication module configured to transmitand receive analog and/or digital signals between elements of system100. For example, communication module 164 may be configured to transmitsensor data from environmental sensor 160 and/or sensor assembly 166 tocommunication module 120 of platform 110 (e.g., for further transmissionto base station 130) or directly to communication module 134 of basestation 130. In other embodiments, communication module 164 may beconfigured to receive control signals (e.g., control signals directingoperation of environmental sensor 160) from controller 112 and/or userinterface 132. In some embodiments, communication module 164 may beconfigured to support spread spectrum transmissions, for example, and/ormultiple simultaneous communications channels between elements of system100.

Sensor assembly 166 may be implemented with one or more sensor elementsupports (e.g., printed circuit boards “PCBs”), connectors, sensorelements, and/or other modules configured to facilitate operation ofenvironmental sensor 160. In a particular embodiment, environmentalsensor 160 may be implemented as a relatively high resolution visiblespectrum camera (e.g., an HD or 2K or 4K visible spectrum camera) andsensor assembly 166 may be implemented as a relatively high resolutionFPA of visible spectrum sensitive detector elements configured togenerate relatively high resolution imagery and/or video of a sceneimaged substantially simultaneously by multispectral imaging system 140.

Orientation and/or position sensor (OPS) 167 of environmental sensor 160may be implemented similar to orientation sensor 114,gyroscope/accelerometer 116, GNSS 118, and/or any other device capableof measuring an orientation and/or position of environmental sensor 160,sensor assembly 166, and/or other elements of environmental sensor 160(e.g., magnitude and direction of roll, pitch, and/or yaw, relative toone or more reference orientations such as gravity and/or MagneticNorth, along with an absolute or relative position) and providing suchmeasurements as sensor signals that may be communicated to variousdevices of system 100.

Power supply 168 may be implemented as any power storage deviceconfigured to provide enough power to each sensor element of sensorassembly 166 to keep all such sensor elements active and able togenerate sensor data while environmental sensor 160 is otherwisedisconnected from external power (e.g., provided by platform 110 and/orbase station 130). In various embodiments, power supply 168 may beimplemented by a supercapacitor so as to be relatively lightweight andfacilitate flight of platform 110 and/or relatively easy handheldoperation of platform 110 (e.g., where platform 110 is implemented as ahandheld sensor platform).

Other modules 170 of environmental sensor 160 may include other and/oradditional sensors, actuators, communication modules/nodes, and/or userinterface devices used to provide additional environmental informationassociated with environmental sensor 160, for example. In someembodiments, other modules 170 may include a humidity sensor, a windand/or water temperature sensor, a barometer, a GNSS, and/or otherenvironmental sensors providing measurements and/or other sensor signalsthat can be displayed to a user and/or used by sensor controller 162 orother devices of system 100 (e.g., controller 112) to provideoperational control of platform 110 and/or system 100 or to processsensor data to compensate for environmental conditions, as describedherein.

In general, each of the elements of system 100 may be implemented withany appropriate logic device (e.g., processing device, microcontroller,processor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), memory storage device, memory reader, orother device or combinations of devices) that may be adapted to execute,store, and/or receive appropriate instructions, such as softwareinstructions implementing a method for providing sensor data and/orimagery, for example, or for transmitting and/or receivingcommunications, such as sensor signals, sensor information, and/orcontrol signals, between one or more devices of system 100.

In addition, one or more non-transitory mediums may be provided forstoring machine readable instructions for loading into and execution byany logic device implemented with one or more of the devices of system100. In these and other embodiments, the logic devices may beimplemented with other components where appropriate, such as volatilememory, non-volatile memory, and/or one or more interfaces (e.g.,inter-integrated circuit (I2C) interfaces, mobile industry processorinterfaces (MIPI), joint test action group (JTAG) interfaces (e.g., IEEE1149.1 standard test access port and boundary-scan architecture), and/orother interfaces, such as an interface for one or more antennas, or aninterface for a particular type of sensor).

Sensor signals, control signals, and other signals may be communicatedamong elements of system 100 using a variety of wired and/or wirelesscommunication techniques, including voltage signaling, Ethernet, WiFi,Bluetooth, Zigbee, Xbee, Micronet, or other medium and/or short rangewired and/or wireless networking protocols and/or implementations, forexample. In such embodiments, each element of system 100 may include oneor more modules supporting wired, wireless, and/or a combination ofwired and wireless communication techniques. In some embodiments,various elements or portions of elements of system 100 may be integratedwith each other, for example, or may be integrated onto a single printedcircuit board (PCB) to reduce system complexity, manufacturing costs,power requirements, coordinate frame errors, and/or timing errorsbetween the various sensor measurements.

Each element of system 100 may include one or more batteries,capacitors, or other electrical power storage devices, for example, andmay include one or more solar cell modules or other electrical powergenerating devices. In some embodiments, one or more of the devices maybe powered by a power source for platform 110, using one or more powerleads. Such power leads may also be used to support one or morecommunication techniques between elements of system 100.

FIG. 2 illustrates a diagram of mobile platforms 110A and 110B ofmultispectral navigation system 200 including embodiments ofenvironmental sensor 160 and associated sensor cradle 128 in accordancewith an embodiment of the disclosure. In the embodiment shown in FIG. 2,multispectral navigation system 200 includes base station 130, optionalco-pilot station 230, mobile platform 110A with articulated imagingsystem/sensor payload 140, gimbal system 122, environmental sensor 160,and sensor cradle 128, and mobile platform 110B with articulated imagingsystem/sensor payload 140, gimbal system 122, environmental sensor 160,and sensor cradle 128, where base station 130 and/or optional co-pilotstation 230 may be configured to control motion, position, orientation,and/or general operation of platform 110A, platform 110B, sensorpayloads 140, and/or environmental sensors 160.

In various embodiments, co-pilot station 230 may be implementedsimilarly relative to base station 130, such as including similarelements and/or being capable of similar functionality. In someembodiments, co-pilot station 230 may include a number of displays so asto facilitate operation of environmental sensor 160 and/or variousimaging and/or sensor payloads of mobile platforms 110A-B, generallyseparate from piloting mobile platforms 110A-B, and to facilitatesubstantially real time analysis, visualization, and communication ofsensor data and corresponding directives, such as to first responders incontact with a co-pilot or user of system 200. For example, base station130 and co-pilot station 230 may each be configured to render any of thedisplay views described herein.

As described herein, embodiments of multispectral navigation system 100may be implemented with a relatively compact, low weight, and low powermultispectral imaging system (e.g., multispectral imaging system 140)that can be used to help operate a mobile platform, such as throughassisted navigation, where processed imagery and/or display views areprovided to an operator to help the operator pilot the mobile platform,or through autopilot navigation, where such imagery is used to autopilotthe mobile platform according to a desired route, destination, or otheroperational parameters.

In some embodiments, a multispectral imaging system may include animaging module implemented by a CMOS based FPA formed, fabricated,assembled, and/or otherwise configured to have sensitivity in the IR,VIS, and UV spectrums/bands. Such imaging module may include a Bayerfilter configured to generate a mosaic or pattern of IR, VIS, and UVpixels in the FPA such that each image captured by the multispectralimaging module includes IR, VIS, and UV information about each sceneimaged by the multispectral imaging module. In particular embodiments,such FPA may be sensitive to portions of the NIR, VIS, and LWUVspectrums, including at least 400-750 nm (VIS), 750-1100 nm (NIR), and330-400 nm (LWUV), and the Bayer filter may be configured to selectivelypass such bands according to a particular mosaic or pattern.

In some embodiments, such Bayer filter may be deposited directly ontothe FPA, for example, and/or may form a checkerboard-like mosaic similarto mosaics used for RGB VIS imaging. Any one of the VIS, NIR, or LWUVspectrums may be emphasized or deemphasized relative to the other twospectrums in resulting imagery according to a particular mosaic selectedfor the Bayer filter and/or according to a particular demosaicingalgorithm (e.g., based on one or more of interpolation, spectralcorrelation, spatial correlation, and/or other demosaicing technique).

In some embodiments, a particular demosaicing algorithm may be selectedbased on one or more environmental conditions associated with an imagedscene or object, for example, or associated with an operational state,position, or orientation of mobile platform 110 and/or imaging module142 of multispectral imaging system 140. For example, a particulardemosaicing algorithm may be configured to deemphasize horizon-alignedpolarized VIS, NIR, and/or LWUV contributions to a captured image basedon a time of day (e.g., a position of the sun in the sky), a position ofmobile platform 110, and/or an orientation of mobile platform 110 and/ormultispectral imaging system 140, so as to reduce image artifacts (e.g.,pixel saturation artifacts) caused by relatively strong reflections ofambient light from horizon-aligned surfaces. In another example, aparticular demosaicing algorithm may be configured to emphasize UVcontributions to a captured image based on a time of day (e.g., a levelof natural ambient light provided by the sun) a position of mobileplatform 110 (e.g., geographical position and altitude to place mobileplatform 110 within a topographical map of the horizon), and/or anorientation of mobile platform 110 and/or multispectral imaging system140, when the UV contribution is expected to relatively low (e.g., whilethe sun is below the horizon).

Such Bayer filter may be implemented as a mosaic of single band passfilters (e.g., each pixel receiving only one of the passed IR, VIS, UVbands), for example, or may be implemented as a mosaic of notchedbroadband transmission filters (e.g., each pixel receiving all but oneof the notched/filtered IR, VIS, UV bands). In embodiments where theBayer filter is implemented as a mosaic of notched broadbandtransmission filters, the selected principal bands may be synthesizedfrom linear combinations of two or more pixels receiving differentiatedspectrums (e.g., associated with spectrally differentiated notchedbroadband transmission filters). In various embodiments, such synthesismay be implemented within/included as part of a demosaicing algorithm,as described herein. Such techniques can provide increased signal tonoise characteristics relative to filter implemented by mosaics ofsingle band pass filters.

In various embodiments, multispectral imaging system 140 is capable ofoperating at reasonably high frame rates so that resulting image streamsare contemporaneous enough to be useful for navigation of a mobileplatform (e.g., where an operator or autopilot often needs to maketime-critical maneuvering decisions). For example, embodiments are ableto operate (e.g., capture and process imagery) at frame ratesapproaching approximately 100 frames/second or higher.

In particular embodiments, multispectral imaging system 140 is capableof cycling an integration time associated with an FPA of imaging module142 over two or more preset values so that multispectral imaging system140 can produce high dynamic range (HDR) imagery in all imaged bands.Such HDR mode may be used to provide midwell exposure values over avariety of lighting conditions, for example, and in some embodiments,the integration times may be determined and/or adjusted by multispectralimaging system 140 based on ambient light level (e.g., in one or morespectrums), contrast levels in prior captured multispectral images,and/or other environmental sensor data and/or derived or processedsensor data and/or imagery. This can be particularly important where,for example, the solar spectrum has very different scene brightnesses inIR, VIS, and UV, and the scene spectrum varies with the diurnal cycle.

A midwell exposure value refers to an exposure event where theintegration capacitor for a sensor element in a CMOS based embodiment ofan FPA of imaging module 142 (e.g., FPA 374 of multispectral imagingsystem 140 of FIG. 3) is allowed to charge to approximately half itscapacity before being read out (e.g., discharged) by module controller372 across PCB 375 (e.g., both of imaging module 142. For example, thereadout frequency and/or the exposure time (e.g., controlled by amechanical, electromechanical, and/or electronic, such LCD, embodimentof shutter 349) may be adjusted based, at least in part, on the averagescene radiance (e.g., of a specific band or across multiple bands, asselected by filter system 376), so that a majority of the sensorelements of FPA 374 associated with one or more bands captured by FPA374 are operating roughly at their midwell exposure values. Operating atsuch midwell exposure values results in image data captured within themost linear portion of the dynamic range of FPA 374 (e.g., providing asubstantially linear response to photons being intercepted by sensorelements of FPA 374), which helps avoid image noise associated with, forexample, low well charge levels. By cycling through different exposuretimes (e.g., integration times), embodiments are able to achieve midwelloperating performance for each of the bands captured by multispectralimaging system 140.

For example, VIS radiance in a scene will often be higher than the NIRor LWUV radiance in the scene. An exposure/integration time of 5milliseconds might provide midwell exposure levels for NIR and LWUVsensor elements (e.g., as selected by filter system 376) but overexposeVIS sensor elements of FPA 374. Multispectral imaging system 140 may beconfigured to capture a first image according to the first 5 millisecondexposure time, then capture a second image according to a second shorterexposure time, and then combine the NIR and LWUV components of the firstimage with the VIS component of the second image to generate an HDRimage (e.g., as long the first and second images are captured one afterthe other and/or while multispectral imaging system 140 or at least FOV345 is substantially stationary with respect to scene 302).

In various embodiments, multispectral imaging system 140 may be equippedwith a lens system that is achromatic across the spectral bands capturedby imaging module 142. Such lens system may be implemented with a focallength chosen to provide a relatively wide field of view (FOV) that issufficient with respect to navigation system and UAS imaging FOVrequirements (e.g., mission, specification, and/or regulatoryrequirements).

In some embodiments, multispectral imaging system 140 may be configuredto process captured imagery according to multispectral image analyticsand/or algorithms (e.g., on-board or after transmission to otherprocessing elements of system 100) configured to classify scene pixelsaccording to their likelihood of being part of a particular class ofobject. For example, clear sky has a distinctive spectrum to it, beingdarker in the NIR and brighter in the UV. Vehicles, even those paintedwhite, tend to have the opposite spectrum: bright in the NIR band anddark in the UV band. Both scene elements can therefore be reliablyclassified based, at least in part, on their spectral characteristics.In various embodiments, such multispectral image analytics and/oralgorithms may be performed by a convolutional neural network (CNN)implemented within multispectral imaging system 140, for example, orwithin one or more controllers associated with multispectral navigationsystem 100.

In particular embodiments, image data provided by multispectral imagingsystem 140 and/or imaging module 142 may be encoded using two bytes perpixel, where 12 bits encode image data (e.g., intensity) and theremaining four bits encode information about a classificationprobability associated with the pixel, such as a 95% probability thepixel is sky or not sky. Such data may then be used by multispectralnavigation system 100 to make maneuvering (e.g., braking and steering)decisions substantially in real time (e.g., at 100+ frames/secondinstances in time).

FIG. 3 illustrates a diagram of multispectral imaging system 140 formultispectral navigation system 100 and/or 300 in accordance with anembodiment of the disclosure. In FIG. 3, multispectral imaging system140 includes imaging module 142 including multispectral FPA 374receiving light 308 from scene 302 through filter system 376, lenssystem 378, and/or optional shutter 349 along optical axis 344 andaccording to FOV 345. In various embodiments, imaging module 142 mayinclude a printed circuit board (PCB) 375 or similar structureconfigured to support FPA 374 and couple FPA 374 and/or other elementsof imaging module 142 to module controller 372 of imaging module 142. Asdescribed herein, filter system 376 may in some embodiments beimplemented as a Bayer filter with a selected mosaic configured toprovide differentiated spectrums (e.g., portions of VIS, IR, and UVspectrums) to pixels of FPA 374. As is also described herein, lenssystem 378 may be achromatic with respect to the differentiatedspectrums provided to pixels of FPA 374, for example, and be configuredto provide FOV 345. In some embodiments, lens system 378 may be actuatedso as to adjust FOV 345, a zoom level of multispectral imaging system140, and/or a focus of light conveyed to FPA 374. In other embodiments,lens system 378 may be a fixed lens system.

Module controller 372 may be implemented as any appropriate processingdevice (e.g., microcontroller, processor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), orother logic device) that may be used by imaging module 142 and/ormultispectral imaging system 140 to execute appropriate instructions,such as software instructions and/or signal processing operations for,for example, capturing multispectral images of scene 302 using FPA 374,filter system 376, lens system 378, and/or shutter 349, demosaicing rawpixel data associated with such multispectral images, and/or classifyingpixels in such images associated with object 304 and/or background 306within scene 302 (e.g., using a CNN implemented within module controller372). Further, module controller 372 may be implemented with varioussignal processing devices, such as analog to digital converters (ADCs),trigger inputs, timing circuitry, and other signal or sensor processingdevices as described herein.

In various embodiments, FPA 374 may be implemented by a two-dimensionalplanar array of similarly fabricated/sized pixel structures eachconfigured to be sensitive across the full spectral band of imagingmodule 142. In other embodiments, FPA 374 may be implemented by an arrayof structurally differentiated pixel structure subarrays, where eachsubarray is sensitive to a differentiated subset of the full spectralband of imaging module 142, for example, and/or may be non-planar (e.g.,concave with respect to optical axis 344), three dimensional (e.g.,multilayered), and/or may include size differentiated pixels (e.g., withlarger surface areas as the distance to optical axis 344 increases).

Filter system 376 may be implemented as a static Bayer filter structuredeposited or otherwise attached to an active surface of FPS 374, forexample, or may be implemented as an adjustable or controllable Bayerfilter structure or other type of filter structure configured to providepixel- or FPA-portion-differentiated spectral illumination of FPA 374.In a specific example, such Bayer filter may be implemented with two VISpixels for each NIR and LWUV pixel (e.g., similar to some Bayer filterpatterns for color VIS cameras with green—2 pixels—and red and blue—1pixel each). Such filters may be implemented as multilayer dielectricinterference-type bandpass filters. More generally, filter system 376may be configured to provide spatially and spectrally differentiatedillumination of FPA 374 according to two or more, or three or moredifferent spectrums, each of which may be full differentiated or maypartially overlap an adjacent differentiated spectrum. In oneembodiment, the characteristics of filter system 376 may include a NIRband of 780-1000 nm, a LWUV band of 370 nm with 60 nm full width halfmaximum (FWHM), and a typical VIS band.

In other embodiments, filter system 376 may be implemented at leastpartially by a multivariate optical element filter array (e.g., amultivariate filter or filter array). Such special interference filtersare typically configured according to relatively complex engineeredtransmission curves that are designed to perform optical computingoperations, such as operations that are analogous to a dot productbetween a scaled regression vector and the spectroscopic response of thescene. For example, a regression vector is commonly a consequence of thedesign of the filter and may be optimized for specific spectra ofinterest. In one embodiments, there such filter may include array with 3or perhaps more distinct multivariate optical elements (MOEs) in aselected pattern. An MOE array designed to specifically detect scenespectra like vegetation, clear sky, overcast sky, road surfaces andvehicles may offer advantages over a simple 3-band approach. Such MOEfilters can be constructed of relatively few layers than conventionalbandpass filters, so they are often physically thinner than bandpassfilters, which makes them an attractive choice for a filter array wherethe pixels of the FPA may have dimensions that are comparable to filterlayer stack thicknesses. Such MOE filters may also tend to have betteroverall scene flux throughput (e.g., when they are composed of fewerlayers than a comparable bandpass filter array).

Lens system 378 may be implemented with one or more lenses eachconfigured to pass light to substantially all pixels of FPA 374, forexample, or may be implemented with an array of lenses (e.g., amicrolens array) each configured to pass light to a subset of pixels ofFPA 374. In general, in embodiments where FPA is sensitive to the NIR,VIS, and LWUV bands, as described herein, each lens of lens system 378may be configured to be color corrected or achromatic from 330-1100 nm.In some embodiments, FOV 345 may be asymmetrical (e.g., to match acorresponding FPA dimension) and be approximately 42 by 34 degrees.

While the embodiment depicted by FIG. 3 shows a relatively compactmultispectral imaging system 140 implemented with a single multispectralimaging module 142 capable of providing single perspective multispectralimagery of scene 302, in other embodiments, multispectral imaging system140 may be implemented with multiple imaging modules 142 each sensitiveto individually differentiated spectrums, for example, and/or eachproviding different perspectives of scene 302, such as according todifferent optical axes and/or different FOVs).

PCB 375 may be a conventional printed circuit board, for example, and beadapted to provide electrical access to FPA 374 and/or other elements ofimaging module 142 (e.g., through various metal traces) as well asphysical support for FPA 374 and/or other elements of imaging module142. In some embodiments, shutter 349 may be implemented as a mechanicalor removable light shield adapted to selectively block one or more bandsof light 308. In various embodiments, shutter 349 may be actuated (e.g.,opened and/or closed) electronically by module controller 372 and/orimaging system controller 312, for example. Shutter 349 may be coupledto/supported by housing 348, for example, and housing 348 may be adaptedto protect system 300 from environmental conditions associated withspace or atmospheric flight, and/or other outdoor environmentalconditions, such as fixed or articulated mounting on a terrestrialvehicle, for example. In other embodiments, housing 348 may be adaptedfor handheld use.

As shown in FIG. 3, multispectral imaging system 140 may be implementedwith a variety of other components adapted to facilitate operation ofmultispectral imaging system 140, including capturing multispectralimages of scene 302, demosaicing images of scene 302, detectingcharacteristics of (e.g., presence, extents, range, translucency,visible color, and/or other characteristics) and/or classifying object304 and background 306 of scene 302 (e.g., as sky or not sky,maneuvering obstruction or not maneuvering obstruction, mobile target ornot mobile target, vegetation or not vegetation, road/earth or notroad/earth, water surface or not water surface, and/or likelihoodthereof), and/or reporting such sensor data to other elements of system100 as described herein. In some embodiments, system 300 may reportsensor data by aggregating sensor data over time (e.g., multiple frames)to provide a time-duration-based reliability of such characteristicsand/or classifications determined by system 300, and then transmittingthe sensor data to other elements of system 100. In other embodiments,system 300 may report sensor data by energizing an LED indicator and/ortransmitting an alert or notification signal to a component (e.g., analarm, or an electrical switch or relay) of systems 300 or 100.

Each of imaging sensor controller 312, memory 146, user interface 332,communication module 144, display 333, and other modules 152, ifoptionally included in multispectral imaging system 140, may be coupledto PCB 375 or to housing 348, for example, depending on a desiredapplication and/or overall size of multispectral imaging system 140and/or imaging module 142. In other embodiments, any one or group ofsuch components may be implemented externally to multispectral imagingsystem 140, for example, and/or in a distributed or grouped manner(e.g., multiple imaging system controllers 312 operating multispectralimaging system 140, or multiple multispectral imaging systems 140operated by a single imaging system controller 312).

Imaging system controller 312 may be implemented as any appropriateprocessing device (e.g., microcontroller, processor, applicationspecific integrated circuit (ASIC), field programmable gate array(FPGA), or other logic device) that may be used by system 300 to executeappropriate instructions, such as software instructions and/or signalprocessing operations for, for example, capturing multispectral imagesof scene 302 using imaging module 142, demosaicing raw pixel dataassociated with such multispectral images, classifying pixels and/orelements of scene 302 in such images (e.g., using a CNN implementedwithin imaging system controller 312), and/or reporting such sensordata/information to other elements of multispectral navigation system100 or 300. Further, imaging system controller 312 may be implementedwith various signal processing devices, such as analog to digitalconverters (ADCs), trigger inputs, timing circuitry, and other signal orsensor processing devices as described herein.

In various embodiments, at least some portion or some functionality ofimaging system controller 312 may be part of or implemented with otherexisting controllers or logic devices of separate systems, such as aserver, a personal electronic device (e.g., a mobile phone, smartphone,tablet device, laptop computer, desktop computer), and/or any otherdevice that may be used to process, report, or act on multispectralimages captured by system 300. In other embodiments, imaging systemcontroller 312 may be adapted to interface and communicate with variousexternal controllers or logic devices and associated components and/orperform various operations in a distributed manner.

In general, imaging system controller 312 may be adapted to interfaceand communicate with other components of system 300 to perform themethods and processes described herein. In one embodiment, imagingsystem controller 312 may be adapted to use communication module 144 toreport multispectral imagery and/or pixel/object classifications todisplay 333 and render and/or display a such information or an alertnotification, for example, or render and/or display an image of aclassification map corresponding to scene 302. In another embodiment,imaging system controller 312 may be adapted to use communication module144 to establish a wired or wireless communication link with a remotereporting system, for example, and report such sensor information.

Memory 146 is typically in communication with at least imaging systemcontroller 312 and may include one or more memory devices (e.g., memorycomponents) to store information, including image data, calibrationdata, other types of sensor data, and/or software instructions. Suchmemory devices may include various types of volatile and non-volatileinformation storage devices, such as RAM (Random Access Memory), ROM(Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory),flash memory, a disk drive, and/or other types of memory. In oneembodiment, memory 146 may include a portable memory device that can beremoved from system 300 and used to convey stored data to other systemsfor further processing and inspection.

Communication module 144 may be configured to facilitate communicationand interfacing between various components of system 300 (e.g., betweenimaging system controller 312 and memory 146 and/or display 333) and/orvarious external devices, such as a wireless access point, a personalelectronic device, a server, and/or other detectors. For example,components such as user interface 332 and display 333 may transmit andreceive data to and from imaging system controller 312 throughcommunication module 144, which may be adapted to manage wired and/orwireless communication links between the various components. As such,communication module 144 may support various interfaces, protocols, andstandards for local system networking, such as the controller areanetwork (CAN) bus, the local interconnect network (LIN) bus, the mediaoriented systems transport (MOST) network, or the ISO 11738 (or ISO bus)standard.

In some embodiments, imaging system controller 312 may be adapted tocommunicate, via communication module 144, with a remote user interface,a notification system, or other detection systems to, for example,aggregate reports from multiple systems or sensors and/or implement aparticular detection and/or notification method. As such, communicationmodule 144 may include a wireless communication component (e.g., basedon the IEEE 802.11 WiFi standards, the Bluetooth™ standard, the ZigBee™standard, or other appropriate short range wireless communicationstandards), a wireless broadband component (e.g., based on WiMaxtechnologies), a mobile cellular component, a wireless satellitecomponent, or other appropriate wireless communication components.Communication module 144 may also be configured to interface with awired network and/or device via a wired communication component, such asan Ethernet interface.

If present, user interface 332 provides for user interaction withmultispectral imaging system 140 and may include one or more buttons,indicators (e.g., LEDs), keyboards, trackballs, knobs, joysticks,displays (e.g., a liquid crystal display, a touch-screen display),and/or other type of user interface adapted to accept user input and/orprovide user feedback. In one embodiment, user interface 332 may includea power button, a vibration motor, an LED to indicate a maneuveringobstruction, and/or a speaker to provide an audible indication of amaneuvering obstruction (e.g., visible, tactile, and/or audibleindicators). In various embodiments, user interface 332 may be used toinput a variety of system configuration settings, such as integrationtime parameters, demosaicing algorithm selections, and/or otherconfiguration settings, as described herein. In some embodiments, userinterface 332 may be used to view one or more reports, graphs, and/orother image data captured by system 300 and/or processed according tothe various operations described herein.

If present, display 333 may be configured to present, indicate, orotherwise convey alerts, notifications, and/or other reports of imagedata and/or object or pixel classifications (e.g., generated by imagingsystem controller 312). Display 333 may be implemented with anelectronic display screen, such as a liquid crystal display (LCD), acathode ray tube (CRT), or various other types of generally known videodisplays and monitors, including touch-sensitive displays. Display 333may be suitable for presenting image data, graphs, video, reports, orother information as described herein.

Other modules 152 may include a temperature sensor/probe (e.g., athermocouple, an infrared thermometer), an LED or laser diode, anambient light sensor, a voltage regulator and/or filter, a variablevoltage source, and/or other types of devices that can be used tofacilitate operation of multispectral imaging system 140, as describedherein. In some embodiments, other modules 152 may include a GNSS,accelerometer, compass, and/or other orientation sensor capable ofsensing a position and/or orientation of multispectral imaging system140. Other modules 152 may additionally include a power moduleimplemented as a battery, a power adapter, a charging circuit, a powerinterface, a power monitor, and/or other type of power supply providinga mobile power source.

In accordance with embodiments described herein, multispectralnavigation systems may benefit from a variety of multispectral imagingand visualization techniques configured to improve the operationalflexibility, reliability, and accuracy of such systems. In particular,embodiments may be configured to provide various display views,including augmented reality views based on imagery provided bymultispectral imaging system 140 and/or other imagers of system 100,allowing a user to access and monitor such features and capabilities,for example, and may be implemented according to various processesand/or control loops configured to ease pilot burden, protect operationof mobile platforms of such systems, and qualitatively andquantitatively evaluate potential maneuvering obstructions and evasionoptions more quickly and more reliably than conventional navigationsystems.

In various embodiments, system 100 may be configured to visualize andcharacterize a maneuvering obstruction through use of multispectralimaging system 140 and other sensors mounted to mobile platform 110. Ingeneral, mobile platform 110 can relay sensor data to an onboardoperator or remote operators at base station 130 and/or co-pilot station230 where the sensor data can be processed or used to maneuver mobileplatform 110. Such sensor data may also be rendered on a display to helpvisualize and characterize the maneuvering obstacle to assist a humanoperator with detecting and evading a maneuvering obstruction. Forexample, elements of system 100 can autonomously map the extents of oneor more maneuvering obstacles and overlay resulting sensor data onto ageospatial chart or imagery, such that an operator can visualize thefull extent of the maneuvering obstacle(s) and proceed safely. Inembodiments where system 100 or 300 includes a handheld mobile platform,elements of system 100 or 300 can aggregate various data to providecritical and timely warnings and/or safety directives to the user of thehandheld platform.

Embodiments may overlay 2D or 3D sensor data onto geospatial maps orimagery as icons or colored highlights or blobs so that users canvisualize the extent of a maneuvering obstacle. Embodiments mayoptionally include a second screen/additional base stations so thatsensor data can be viewed/processed by a user other than a UAV/UGVpilot.

In some embodiments, a display view (e.g., rendered by user interface132 and/or display 333) may include a geospatial chart or augmentedimagery surrounded by various selector/indicator groups (e.g., a header,payload controller menus, video feed, and platform telemetry indicator)configured to visualize and/or quantify maneuvering obstacles andoperate mobile platform 110 and/or elements of mobile platform 110. Forexample, a header may include one or more selectors and/or indicatorsconfigured to receive user selection of a particular selector to enable,disable, or select active sensor payloads (e.g., multispectral imagingsystem 140, environmental sensor 160) for display of correspondinggeoreferenced sensor data within a geospatial chart or augmentedimagery, for example, or to indicate an operational status of mobileplatform 110 and/or various elements of mobile platform 110.

In related embodiments, a geospatial chart or augmented imagery includesa mobile platform indicator and a maneuvering obstacle overlay renderedover a base map or chart. In various embodiments, system 100 may beconfigured to determine a shape, extent, and/or other characteristics ofa maneuvering obstacle overlay within the geospatial chart or augmentedimagery based, at least in part, on sensor data provided bymultispectral imaging system 140, environmental sensor 160, andorientation and/or position data (e.g., provided by OPS 167 and/or otherorientation and/or position or motion sensors of mobile platform 110 orelements of mobile platform 110) as mobile platform 110 maneuvers withinthe area shown in the geospatial chart or augmented imagery. Forexample, system 100 may be configured to determine an extent associatedwith object 304 from the perspective of optical axis 344 based on sensordata and/or environmental conditions provided by mobile platform 110,and render the maneuvering obstruction overlay according to a colormapping to indicate relative range or approaching velocity, such as hotcolors (e.g., red) to indicate relatively close or quickly approachingmaneuvering obstructions, and cold colors (e.g., blue) to indicaterelatively far or quickly receding maneuvering obstructions.

In another embodiment, system 100 may be configured to determinemultiple types of maneuvering obstacles are present within a particularsurvey area or scene, for example, and render each type of maneuveringobstacle according to a different overlay layer presented in a displayview, each of which may be selective enabled and/or disabled by a user.

In various embodiments, mobile platform 110 may be configured to adjustits course based on sensor data provided by multispectral imaging system140 and/or environmental sensor 160, for example, and/or based onvarious environmental conditions measured by sensors mounted to mobileplatform 110 or by external systems and communicated to system 100(e.g., such as regional weather data provided by an online database overa wireless network linked to base station 130 or co-pilot station 230).As such, mobile platform 110 may be configured to autonomously avoidmaneuvering obstacles or hazardous environments (e.g., significantdowndrafts or otherwise undesirable environmental conditions and/ormaneuvering obstacles within such undesirable environmental conditions).For example, sending a UAV/UGV into a hazardous environment can putmobile platform 110 at risk of damage. By adding intelligent maneuveringobstacle avoidance based on multispectral imagery and environmentalsensors carried on-vehicle, risk of collision and/or inefficientmaneuvering can be limited through automatic course adjustment, therebyprotecting mobile platform 110 and it associated sensor suite.

Embodiments described herein may provide for autonomous reaction tomaneuvering obstacles. For example, controller 112 and/or a controllerof base station 130 or co-pilot station 230 may be configured to receivemultispectral imagery, classification data, and/or other sensor datafrom mobile platform 110 and/or from sensors mounted to mobile platform110 and to determine course adjustments to avoid detected maneuveringobstacles and/or environmental conditions. Examples of courseadjustments may include halt, divert around, climb, and/or reversecourse to retreat from or otherwise avoid a maneuvering obstacle ordangerous environment. Such course adjustments may be relayed to a userof base station 130, for example, or may be implementeddirectly/autonomously by mobile platform 110. Such autonomous responseis intended to preserve the integrity of mobile platform 110 andfacilitate reaching a desired destination.

FIGS. 4-6 illustrate display views 400, 500, and 600 including imagerygenerated by multispectral imaging system 140 for multispectralnavigation system 100 or 300 in accordance with embodiments of thedisclosure. In FIG. 4, display view 400 shows a relatively highresolution visible spectrum RGB or color image 402 of a neighborhood andvarious scene elements (e.g., a road, sidewalk, fence, vegetation, and astructure behind the vegetation, with all scene elements beneath acloudy sky). In FIG. 5, display view 500 shows a monochrome visiblespectrum image 502, a NIR image 504, and a LWUV image 506 of the samescene depicted in visible spectrum color image 402. FIG. 6 shows displayview 600 including multispectral image 602 including the spectralcharacteristics of each of monochrome visible spectrum image 502, NIRimage 504, and LWUV image 506, where each differentiated spectrum ismapped into the R, G, and B channels typically visible by human eyes(e.g., NIR image data is mapped to the R channel, VIS data is mapped tothe G channel, and LWUV data is mapped to the Blue channel).

As can be seen from FIGS. 4-6, the daytime sky spectral signature has arelatively low spectral signature/brightness in NIR imagery, mediumspectral signature/brightness in VIS imagery, and relatively very brightspectral signature/brightness in LWUV imagery. As such, objectssilhouetted against the sky in LWUV imagery are much easier to segmentout from the sky than when using VIS imagery, and multispectral analysisis better able to discriminate between sky and foreground objects, suchas other mobile platforms. For example, it is very hard to envision adaytime scenario where both the sky and some closing object have thesame apparent radiance in all three of the bands depicted in FIG. 5,simultaneously. Moreover, such multispectral analysis is particularlyuseful when an object or maneuvering obstacle is beyond a reliable LIDARrange. As such, embodiments are typically able to use simple imagesubtraction to segment out the sky from other objects in the scene, evenwhen the object look similar in the visible spectrum. Furthermore, anon-board CNN or other machine vision engine (e.g., implemented withinmodule controller 372 and/or imaging system controller 312) couldperform pixel and/or object classification rapidly and sendmultispectral image data including “tagged” pixels or groups of pixelsto elements of multispectral navigation system 100 and determine variousmaneuvering adjustments to avoid maneuvering obstacles, as describedherein.

In the various images of FIG. 5, it can be seen that natural lightshadows are suppressed in the UV band. For example, Rayleigh scatteringtypically makes the whole sky glow relatively brightly with scattered UVsunlight. Since the whole sky dome is lit, shadows are less intense inthe UV band (e.g., as shown in LWUV image 506). Moreover, manyforeground objects in LWUV imagery will tend to look dark, since UV isabsorbed by many molecular surfaces. By contrast, shadows tend to beenhanced in the NIR band because there is less Rayleigh scattering(e.g., as shown in NIR image 504). Therefore, subtracting NIR image 504and LWUV image 506 from each other results in a multispectral imageemphasizing which pixels in the image are likely to be shadows (e.g.,thereby classifying such pixels as shadow or not shadow, optionally withan associated likelihood).

In another example, white clothing against clean snow is typicallyinvisible or low contrast in VIS imagery, particularly in diffuseambient light with indistinct or absent shadowing. However, whiteclothing against clean snow in LWUV imagery is typically very pronouncedwith relatively high contrast and can be detected fairly easily with CNNanalysis and/or image subtraction of the LWUV imagery from the VISimagery.

FIG. 7 illustrates a flow diagram 700 of various operations to provideassisted or autopiloted navigation using a multispectral navigationsystem in accordance with embodiments of the disclosure. In someembodiments, the operations of FIG. 7 may be implemented as softwareinstructions executed by one or more logic devices or controllersassociated with corresponding electronic devices, sensors, and/orstructures depicted in FIGS. 1-3. More generally, the operations of FIG.7 may be implemented with any combination of software instructions,mechanical elements, and/or electronic hardware (e.g., inductors,capacitors, amplifiers, actuators, or other analog and/or digitalcomponents).

It should also be appreciated that any step, sub-step, sub-process, orblock of process 700 may be performed in an order or arrangementdifferent from the embodiments illustrated by FIG. 7. For example, inother embodiments, one or more blocks may be omitted from or added toeach individual process. Furthermore, block inputs, block outputs,various sensor signals, sensor information, calibration parameters,and/or other operational parameters may be stored to one or morememories prior to moving to a following portion of a correspondingprocess. Although process 700 is described with reference to systemsdescribed in FIGS. 1-3, process 700 may be performed by other systemsdifferent from those systems and including a different selection ofelectronic devices, sensors, assemblies, mechanisms, platforms, and/orplatform attributes.

Process 700 of FIG. 7 may generally correspond to a method fornavigating a survey area using multispectral navigation system 100.

At block 702, multispectral image data corresponding to a projectedcourse for a mobile platform is received. For example, controllers 112,312, and/or 372, communication modules 120, 144, and/or 134, userinterface 132, and/or other elements of system 100 may be configured toreceive multispectral image data from multispectral imaging system 140and/or imaging module 142 as mobile platform 110 maneuvers along aprojected course (e.g., within scene 302).

In block 704, orientation and position data corresponding tomultispectral image data is received. For example, system 100 may beconfigured to receive orientation and/or position data (e.g., fromvarious orientation, position, and/or other motion sensors of system100) corresponding to the multispectral image data received in block702.

In block 706, maneuvering obstacle information is generated. Forexample, system 100 may be configured to generate maneuvering obstacleinformation (e.g., indicating a position, extent, and/or othercharacteristics of object 304 in scene 302) corresponding to theprojected course of mobile platform 110 (e.g., within scene 302) based,at least in part, on a combination of the orientation and/or positiondata and the multispectral image data received in blocks 702 and 704.

In block 708, a display view including maneuvering obstacle informationis rendered. For example, system 100 may be configured to render adisplay view (e.g., display views of FIGS. 4-6) including themaneuvering obstacle information generated in block 706 in a display ofuser interface 132 and/or in display 333 of multispectral imaging system140.

In block 710, intersection of a projected course with a maneuveringobstacle area is determined. For example, system 100 may be configuredto determine the projected course for mobile platform 110 intersects aposition and/or extent of at least one object 304 in scene 302 based, atleast in part, on the maneuvering obstacle information generated inblock 706.

In block 712, a projected course of a mobile platform is adjusted. Forexample, system 100 may be configured to adjust the projected course ofmobile platform 110 to avoid one or more maneuvering obstacles (e.g.,multiple objects 304 in scene 302) intersecting the projected course formobile platform 110 as determined in block 710. For example, system 100may be configured to determine an avoidance course configured to avoidall maneuvering obstacles within scene 302 and to reach a predetermineddestination or traverse scene 302 generally according to a predeterminedheading or approach. In other embodiments, system 100 may be configuredto determine a series of avoidance courses configured to avoidindividual maneuvering obstacles within scene 302 as mobile platform 110maneuvers through scene 302.

By providing such systems and techniques for multispectral navigation,embodiments of the present disclosure substantially improve theoperational flexibility and reliability of manned and unmanned mobileplatforms, including unmanned sensor platforms. Moreover, such systemsand techniques may be used to increase the operational safety of usersand operators of mobile platforms, including of unmanned mobile sensorplatforms, beyond that achievable by conventional systems. As such,embodiments provide multispectral imaging systems and navigation systemswith significantly increased operational convenience and performance.

As noted above, another important class of object in a scene that iscommonly encountered by vehicles is vegetation. Healthy vegetationstrongly reflects NIR radiation, especially in the 800 nm band. A camerasystem with the ability to measure both visible-band radiance and NIRradiance may be configured to detect the so-called Red Edge: the sharprise in reflectivity from 700 nm to 800 nm associated with spongymesophyll tissue in most vegetation.

One algorithm for identifying foliage is the normalized differentialvegetative index or NDVI. This metric is used quite often in remotesensing from satellites. The traditional NDVI is most commonly definedas the normalized contrast between the NIR band and the visible red bandin multispectral images. With respect to embodiments of the disclosedmultispectral imaging system, there often is no separate visible redband, as distinct from visible green or blue, and so the traditionalNVDI may be modified to form the mNDVI, to define it according to thecontrast between NIR and full visible spectrum light:

mNDVI=(NIR−VIS)/(NIR+VIS)

Using this definition of the mNDVI, thresholds can be identified andselected to classify pixels in a multispectral image as associated withvegetation in the imaged scene. A typical range of threshold valuesincludes mNDVIs of 0.3-0.4.

Another useful metric may be referred to as the normalized differentialsky index or NDSI. For example, there is often strong contrast betweenLWUV and NIR images of the sky because the Rayleigh scattering crosssection varies very strongly with wavelength:

σ_(Rayleigh)˜Wavelength⁻⁴

The LWUV light will be scattered approximately sixteen times more thanNIR light (e.g., with twice the wavelength), which makes the sky appearbright in the LWUV band and dark in the NIR band. This NDSI metric maybe defined as:

NDSI=(LWUV−NIR)/(LWUV+NIR)

Using this definition of the NDSI, thresholds can be identified andselected to classify pixels in a multispectral image as associated withsky in the imaged scene. A typical threshold value includes an NDSI ofapproximately 0.2.

FIGS. 8-10 illustrate display views 800, 900, and 1000 including imagerygenerated by multispectral imaging system 140 for multispectralnavigation system 100 or 300 in accordance with embodiments of thedisclosure. FIGS. 8-10 show three views of the same scene. In FIG. 8,display view 800 shows a relatively high resolution visible spectrum RGBor color image 802 of a major intersection with a highway onramp andvarious scene elements (e.g., streetlights, a road with painted lane anddirection indicators, street signs, sidewalks, fencing, vegetation, abridge, and a mountain range, with all scene elements beneath a clearsky). More generally, FIG. 8 shows a full color visible-light image of atypical ADAS scene.

In FIG. 9, display view 900 shows a multispectral image 902 includingthe spectral characteristics of each of a VIS image, a NIR image, and aLWUV image of the same scene depicted in visible spectrum color image802, where each differentiated spectrum is mapped into the R, G, and Bchannels typically visible by human eyes (e.g., NIR image data is mappedto the R channel, VIS data is mapped to the G channel, and LWUV data ismapped to the Blue channel). FIG. 10 shows display view 1000 includingprocessed image or classification map 1002 that shows vegetation in red,sky in blue, and the remainder in black, as identified pixel by pixelusing the mNDVI and the NDSI, along with appropriate thresholds, asdescribed herein. For example, the specific mNDVI and NDSI thresholdsused to generate classification map 1002 are 0.35 and 0.2, respectively.

Embodiments of the disclosed multispectral imaging system may beconfigured to differentiate green-colored objects, such as green roadsigns, from green vegetation. Such ability makes it easier for an ADASto identify and segment out green road signs and use optical characterrecognition to incorporate the information in imaged text into itsgeneral data streams. Moreover, a green-colored vehicle is easier to seeagainst green vegetation using measurements from the NIR and VIS bands.By contrast, both green signs and green-colored vehicles on a road orparked next on the side of a road may be at risk of being lost against abackdrop of green vegetation if a conventional color camera is used.

For example, a LWUV image of a highway may provide minimal contrastbetween a sign and vegetation behind it, but, in the LWUV image, thereis typically relatively high contrast between the sky and everythingelse. A multispectral image of the same scene would therefore be able toshow the sky and vegetation clearly delineated from each other. With thespectral mappings provided herein, a road might be depicted in amultispectral image with a yellow-grey color. As such, it is possible toclassify the road surface (e.g., using mapped RGB color thresholds) asdistinct from both vegetation and sky in the imaged scene, since themultispectral appearance of the road surface is substantially differentfrom that of the sky and the vegetation. Selecting appropriatethresholds, structural morphologies, and/or identifying otherclassification processing characteristics or techniques may includeimplementing an appropriate CNN training and classification technique,where the CNN is trained to classify various different image featuresimportant to an ADAS.

Using the techniques described herein, embodiments of the multispectralnavigation system described herein are able to: identify vegetation,since it is bright in the NIR band, but darker in the other two bands;identify clear sky, since it is bright in the LWUV band, but darker inthe other two bands; distinguish red LED taillights from incandescenttaillights with red filters; and define the location of car windows bythe visible light passing through them. Embodiments are also able to:differentiate man-made surfaces from natural surfaces; differentiategreen vehicles from vegetation; differentiate sky-blue vehicles fromclear sky; differentiate white vehicles from overcast sky; differentiateicy roads from ice-free roads; and differentiate wet roads from dryroads.

In 2019, vehicle accidents in the United States killed more than 6,000pedestrians, the highest annual total ever recorded, and sent more than100,000 to hospitals with injuries. As the automotive industry movestowards autonomous vehicles (AV), the ability to sense, classify, andmake split-second artificial intelligence (AI) based maneuveringdecisions while driving becomes increasingly necessary. Advanced driverassistance systems (ADAS) and related AV systems are tasked withbecoming smarter and safer quickly. Embodiments described herein offerautomotive manufacturers, suppliers, regulators, automotive testingagencies, commercial vehicle operators, and consumers systems thatmaximize safety of drivers, pedestrians, and other vulnerable roadusers.

FIG. 11 is a diagram 1100 illustrating the functional benefitsassociated with thermal imaging navigation systems (e.g., embodiments ofmultispectral navigation system 100 in FIG. 1) in accordance with anembodiment of the disclosure. In particular, diagram 1100 of FIG. 11shows how thermal imaging-based navigation systems can providerelatively reliable feature performance over a relatively large portionof the safety feature phase space identified in diagram 1100. Moreover,diagram 1100 of FIG. 11 shows that increased reliable featureperformance can be achieved by combining thermal imaging with visiblespectrum imaging and/or other remote sensor systems (e.g., radar), wherethe overlap or fusion of the different feature performancessubstantially fills the safety feature phase space identified in diagram1100.

FIG. 12 illustrates a diagram of a mobile platform 110 employing athermal imaging navigation system 1200 in accordance with an embodimentof the disclosure. For example, as described herein, in someembodiments, multispectral navigation system 100 may be implemented as athermal imaging navigation system, where sensor payload 140 may beimplemented as a thermal imaging system 140 including a thermal imagingmodule 142, environmental sensor 160 may be implemented as a visiblespectrum imaging system 160 including a visible spectrum imagingmodule/sensor assembly 166, and thermal imaging navigation system 100may include ranging sensor system 127, which may be implemented as aradar or other type of ranging sensor system. In such embodiments, eachof thermal imaging system 140, visible spectrum imaging system 160, andranging sensor system 127 may be mounted to platform 110 so as to haveoverlapping fields of view (e.g., overlapping sensor data of scene 302).

In particular, thermal imaging navigation system 1200 of FIG. 12 mayinclude one or more of controller 112, propulsion system 124 (e.g., anelectric motor or combustion engine or hybrid motor, coupled to atransmission and/or drive train), braking system 125 (e.g., one or moreelectromechanically controlled clamping or motion retardation devicesdisposed along the drivetrain of propulsion system 124, including at orwithin wheels for platform/passenger vehicle 110), ranging sensorsystems 127 a (a grille mounted radar system), 127 b (a front bumpermounted radar or sonar system), 127 c (a pillar mounted radar system),and/or 127 d (a rear bumper or trunk mounted radar or sonar system),thermal imaging systems 1240 a (a roof mounted “shark fin” or “hat” orradio antenna-integrated thermal imaging system), 1240 b (a pillarmounted thermal imaging system), 1240 c (a rear windshield mountedthermal imaging system, and/or 1240 d (a front windshield mountedthermal imaging system), and/or visible spectrum imaging system 1260 a(a front windshield mounted visible spectrum imaging system), 1260 b (aroof or roof-rack 1226 mounted visible spectrum imaging system), 1260 c(a rear windshield mounted visible spectrum imaging system), and/or 1260d (a trunk mounted visible spectrum imaging system).

More generally, each of the mounting spots identified in FIG. 12 may beused to mount any one or combination of a thermal imaging system, avisible spectrum imaging system, and/or a remote sensor system, asdescribed herein. In various embodiments, all sensor data generated byeach of the mounted systems may be used to generate display viewsrendered by user interface 1232 of platform 110 (e.g., a dash displayfor passenger vehicle 110).

FIG. 13A illustrates a data flow diagram 1300 of mobile platform 110employing thermal imaging navigation system 1200 in accordance with anembodiment of the disclosure. In particular, data flow diagram 1300shows thermal imaging system 1260 and/or visible spectrum imaging systemproviding thermal and/or visible spectrum imagery to maneuveringobstacle detector 1340, which may be configured to provide the receivedimagery and/or associated maneuvering obstacle information (e.g., taggedimagery) generated by maneuvering obstacle detector 1340 to rangeestimator 1342. Range estimator 1342 may be configured to determine andprovide a range estimate associated with each detected maneuveringobstacle represented in the received tagged imagery to sensor data fusor1344, and sensor data fusor 1344 may be configured to use ranging sensordata provided by ranging sensor system 127 and/or orientation, position,motion, and/or other registration or calibration data provided byregistrator 1318 to fuse or otherwise combine the tagged imagerygenerated by maneuvering obstacle detector 1340 and the associated rangeestimates, for example, and/or the ranging sensor data provided byranging sensor system 127, as shown.

Sensor data fusor 1344 may be configured to provide the combined sensordata and/or imagery to braking planner 1346 (e.g., an automaticemergency braking planner), which may be configured to evaluate thecombined sensor data and/or imagery, including a projected course ofplatform 110 (e.g., provided by registrator 1318) and selectivelyactivate braking system 125 and/or other elements of propulsion system124 to avoid colliding with any of the maneuvering obstacles detected bymaneuvering obstacle detector 1340.

In optional embodiments, maneuvering obstacle detector 1340 may beconfigured to generate tagged imagery based on any one or combination ofthermal imagery, visible spectrum imagery, and/or ranging sensor data.Moreover, range estimator 1342 may be configured to determine andprovide range estimates associated with each detected maneuveringobstacle based on the tagged imagery provided by maneuvering obstacledetector 1340 and/or ranging sensor data provided by ranging sensorsystem 127.

In various embodiments, each of maneuvering obstacle detector 1340,range estimator 1342, sensor data fusor 1344, and/or braking planner1346 may be implemented and/or executed as individual software programsby controller 112. In particular embodiments, maneuvering obstacledetector 1340 may be implemented as one or more CNNs configured togenerate tagged thermal, visible, and/or blended imagery, such asthrough one or more of feature extraction, sematic segmentation, objectrecognition, classification, and/or other similar CNN based image and/orsensor data processing.

In one embodiment, maneuvering obstacle detector 1340 may be configuredto apply a thermal imagery trained CNN to detect maneuvering obstaclesrepresented in the thermal imagery provided by thermal imaging system1240 and generate corresponding tagged thermal imagery and/or associatedmap scores (e.g., accuracy likelihood values) for each maneuveringobstacle detected in the thermal imagery. In a related embodiment,maneuvering obstacle detector 1340 may be configured to apply a visiblespectrum imagery trained CNN to detect maneuvering obstacles representedin the visible spectrum imagery provided by visible spectrum imagingsystem 1260 and generate corresponding tagged visible spectrum imageryand/or associated map scores for each maneuvering obstacle detected inthe visible spectrum imagery. In such embodiments, maneuvering obstacledetector 1340 may be configured to combine the two sets of tagged imagesaccording to a logic function, such as according to one or the otheridentifying a maneuvering obstacle with a map score above aspectrum-specific threshold value, and/or any commonly detectedmaneuvering obstacle with a combined map score (from each spectrum)above a combined threshold value.

In another embodiment, maneuvering obstacle detector 1340 may beconfigured to blend the thermal imagery with the visible spectrumimagery prior to applying a blended imagery trained CNN to detectmaneuvering obstacles represented in the blended imagery and generatecorresponding tagged blended imagery and/or associated map scores foreach maneuvering obstacle detected in the blended imagery, where onlymaneuvering objects with map scores above a blended imagery thresholdvalue are forwarded as tagged blended imagery to range estimator 1342.In a further embodiment, maneuvering obstacle detector 1340 may beconfigured to blend the thermal imagery with the visible spectrumimagery and combine the result with ranging sensor data provided byranging sensor system 127 prior to applying a fused sensor data trainedCNN to detect maneuvering obstacles represented in the fused sensor dataand generate corresponding tagged blended or spectrum-specific imageryand/or associated map scores for each maneuvering obstacle detected inthe fused sensor data, where only maneuvering objects with map scoresabove a fused sensor data threshold value are forwarded as taggedimagery to range estimator 1342. In various embodiments, any of the CNNsdescribed herein may be trained via synthetic maneuvering obstacles,where computer generated animals and/or other maneuvering obstacles areadded to imagery in order to train the CNNs to tag associated imageryappropriately.

Thermal and visible spectrum imagery blending may be performed accordingto a variety of metrics emphasizing one or more characteristics of oneor the other spectrums. In some embodiments, color characteristics ofthe visible spectrum imagery may be modulated according to overlappingthermal imagery pixel values. In other embodiments, the pixel values ofthe visible spectrum imagery may be converted to greyscale before beingblended with overlapping thermal imagery pixel values. In furtherembodiments, the thermal imagery pixel values may be mapped to aparticular color palette before blended with overlapping visiblespectrum imagery pixel values (e.g., greyscale or color pixel values).

In some embodiments, range estimator 1342 may be configured to generatea range and/or relative direction estimate based only on thermal orvisible spectrum imagery, such as by identifying common object featureswith known average feature displacements (e.g., port and starboard taillights) and determining the range estimate based on the known averagefeature displacement, the pixel displacement of the identified objectfeatures, and one or more displacement calibration parameters (e.g.,generally specific to each imaging system). In other embodiments, rangeestimator 1342 may be configured to generate a range and/or relativedirection estimate based on one or any combination of thermal imagery,visible spectrum imagery, or ranging sensor data provided by rangingsensor system 127. For example, if each imaging system is known andfixed relative to ranging sensor system 127, such range estimate may beperformed without registration data provided by registrator 1318.

Sensor data fusor 1344 may be configured to fuse substantiallysynchronous sensor data provided by any of the sensors of systems 100and/or 1200, for example, and/or may be configured to fuse temporallydifferentiated data, such as a time series of sensor data and/or taggedimages, so as to facilitate accurate maneuvering obstacle tracking, asdescribed herein. Braking planner 1346 may be configured to receive allthe individual sensor data, tagged imagery, range and/or relativedirection estimates, and/or fused sensor data and selective activatebraking system 125 and/or other elements of propulsion system 124 tohalt or reduce a speed of platform 110 to avoid potential collision witha maneuvering obstacle detected by maneuvering obstacle detector 1340,as described herein.

FIG. 13B illustrates a block diagram of an update system 1302 for mobileplatforms 110 employing thermal imaging navigation systems 1200 inaccordance with an embodiment of the disclosure. As can be seen in FIG.13B, update system 1302 may include various platforms 110 eachconfigured to receive CNN configurations (weights) from update server1390 via network 1320. Each of platforms 110 (e.g., passenger vehicles)may be implemented as described with respect to platform 110 of FIGS. 1and/or 12. In various embodiments, communication network 1320 may beimplemented according to one or more wired and/or wireless networkinterfaces, protocols, topologies, and/or methodologies, as describedherein, and in some embodiments may include one or more LAN and/or WANnetworks, including cellular networks and/or the Internet.

In some embodiments, CNN maneuvering obstacle detection failures may beidentified (e.g., by user override of thermal imaging navigation system1200) and/or stored locally at platform 110. In some embodiments, updateserver 1390 may be configured to receive CNN maneuvering obstacledetection failures and associated sensor data and generate updated CNNconfigurations to compensate for such edge cases/failures. For example,update server 1390 and/or a connected dedicated CNN trainer and/or anannotation feedback loop may be configured to adjust a current CNNconfiguration based, at least in part, on the CNN maneuvering obstacledetection failure and may store the updated CNN configuration fordistribution to platforms 110.

Update server 1390 may be implemented as a logic device, a tabletcomputer, laptop, desktop, and/or server computer that may be configuredto implement a CNN configuration database that stores and manages CNNconfigurations associated with different platforms and provide updatedCNN configurations to platforms 110 when requested by users, forexample, or when pushed by a manufacturer or regulator. Although network1320 is shown as one element in FIG. 13B, in various embodiments,network 1320 may include multiple network infrastructures and/orcombinations of infrastructures where, for example, each platform 110may be configured to use substantially different network infrastructuresto access update server 1390.

FIGS. 14A-B illustrate display views 1400, 1402 including imagerygenerated by thermal imaging system 1240 for thermal imaging navigationsystem 1200 in accordance with embodiments of the disclosure. Forexample, display view 1400 of FIG. 14A shows visible spectrum image1460, co-registered thermal image 1440 including image tagging (e.g.,CNN based maneuvering obstacle image tags 1480 and radar sensor datatags 1482), and a top-down or birds-eye view of the tagged imagery asfused image 1427, which includes CNN based maneuvering obstacle imagetags 1480, radar sensor data tags 1482, and a local area image tagassociated with a position of platform 110. Display view 1402 of FIG.14B shows visible spectrum image 1462 with visible spectrum imagery CNNbased maneuvering obstacle tag 1486 (e.g., a person), thermal image 1442with thermal imagery CNN based maneuvering obstacle tag 1484 (e.g., thesame person), and blended image 1444 with blended imagery CNN basedmaneuvering obstacle tag 1488.

FIG. 15 illustrates a flow diagram 1500 of various operations to provideassisted or autopiloted navigation including automated emergencybreaking using a thermal imaging navigation system in accordance withembodiments of the disclosure. In some embodiments, the operations ofFIG. 15 may be implemented as software instructions executed by one ormore logic devices or controllers associated with correspondingelectronic devices, sensors, structures, and/or related imagery ordisplay views depicted in FIGS. 1-14B. More generally, the operations ofFIG. 15 may be implemented with any combination of softwareinstructions, mechanical elements, and/or electronic hardware (e.g.,inductors, capacitors, amplifiers, actuators, or other analog and/ordigital components).

Any step, sub-step, sub-process, or block of process 1500 may beperformed in an order or arrangement different from the embodimentsillustrated by FIG. 15. For example, in other embodiments, one or moreblocks may be omitted from or added to each individual process.Furthermore, block inputs, block outputs, various sensor signals, sensorinformation, calibration parameters, and/or other operational parametersmay be stored to one or more memories prior to moving to a followingportion of a corresponding process. Although process 1500 is describedwith reference to systems described in FIGS. 1-14B, process 1500 may beperformed by other systems different from those systems and including adifferent selection of electronic devices, sensors, assemblies,mechanisms, platforms, and/or platform attributes.

Process 1500 of FIG. 15 may generally correspond to a method fornavigating a roadway and braking to avoid maneuvering obstacles using athermal imaging navigation system 1200 (e.g., an embodiment ofmultispectral navigation system 100 of FIG. 1).

At block 1502, thermal image data corresponding to a projected coursefor a mobile platform is received. For example, controllers 112, 312,and/or 372, communication modules 120, 144, and/or 134, user interfaces1232 and/or 132, and/or other elements of systems 100 and/or 1200 may beconfigured to receive thermal image data from thermal imaging system1240 and/or imaging module 142 as mobile platform 110 maneuvers along aprojected course (e.g., within scene 302).

At block 1504, sensor data corresponding to thermal image data isreceived. For example, system 1200 may be configured to receive visiblespectrum image data, radar data, lidar data, other ranging sensor data,and/or orientation and/or position data (e.g., from various orientation,position, and/or other motion sensors of system 100) corresponding tothe thermal image data received in block 1502. In some embodiments,system 1200 may be configured to blend thermal and visible spectrumimagery prior to proceeding to block 1506, for example, or otherwisecombining the thermal and visible spectrum imagery with received rangingsensor data prior to proceeding to block 1506, as described herein. Insome embodiments, system 1200 may be configured to adjust a framerate ofany of the imaging systems of system 1200 based, at least in part, on aspeed of platform 110.

At block 1506, maneuvering obstacle information is generated. Forexample, system 1200 may be configured to generate maneuvering obstacleinformation (e.g., indicating a position, extent, and/or othercharacteristics of object 304 in scene 302) corresponding to theprojected course of mobile platform 110 (e.g., within scene 302) based,at least in part, on the thermal image data received in block 1502. Inother embodiments, system 1200 may be configured to generate maneuveringobstacle information corresponding to the projected course of mobileplatform 110 based, at least in part, on a combination of the sensordata and the thermal image data received in blocks 1502 and 1504, asdescribed herein. In various embodiments, such maneuvering obstacleinformation may include tagged thermal, visible spectrum, and/or blendedimagery, as described herein. In some embodiments, such maneuveringobstacle information may include range and/or relative directionestimates, fused sensor data corresponding to detected maneuveringobstacles, and/or other sensor data or processed sensor data, asdescribed herein.

At block 1508, a display view including maneuvering obstacle informationis rendered. For example, system 1200 may be configured to render adisplay view (e.g., display views of FIGS. 13A-14B) including themaneuvering obstacle information generated in block 1506 in a display ofuser interface 1232 and/or in a display of user interface 132. Suchdisplay view may include visible spectrum imagery, thermal spectrumimagery, blended imagery, and/or a fused sensor data, for example,and/or one or more types of image tagging, as described herein.

At block 1510, intersection of a projected course with a maneuveringobstacle area is determined. For example, system 1200 may be configuredto determine the projected course for mobile platform 110 intersects aposition and/or extent of at least one object 304 in scene 302 based, atleast in part, on the maneuvering obstacle information generated inblock 1506. For example, each of such determined intersections may bedetermined by braking planner 1346 and/or controller 112, as describedherein.

At block 1512, a braking system of a mobile platform is activated. Forexample, system 1200 may be configured to activate braking system 125and/or control other elements of propulsion system 124 of mobileplatform 110 to halt or reduce motion of platform 110 to avoid one ormore maneuvering obstacles (e.g., multiple objects 304 in scene 302)intersecting the projected course for mobile platform 110, as determinedin block 1510. For example, system 1200 may be configured to determinean avoidance course configured to avoid all maneuvering obstacles withinscene 302 while braking platform 110 without losing steering control ofplatform 110 and generally according to a predetermined heading orapproach. In other embodiments, system 1200 may be configured todetermine a series of avoidance courses configured to avoid individualmaneuvering obstacles within scene 302 as mobile platform 110 brakes.More simply, system 1200 may be configured to determine a breaking powerto be applied to braking system 125 to halt platform 110 in the leastamount of linear travel possible.

By providing such systems and techniques for thermal image-basednavigation, embodiments of the present disclosure substantially improvethe operational flexibility and reliability of manned and unmannedmobile platforms, including passenger vehicles. Moreover, such systemsand techniques may be used to increase the operational safety of usersand operators of mobile platforms, including of manned passengervehicles, beyond that achievable by conventional systems. As such,embodiments provide thermal imaging navigation systems withsignificantly increased operational convenience and performance.

As described herein, embodiments present an elegant software solutionthat combines visible spectrum imagery and/or video with thermal imageryand/or video to create a blended combination of the two. In someembodiments, such blended imagery uses partial thermal imagery and alsovisible spectrum imagery to provide features like color and the abilityto read signs and lane markings, even in low light conditions. Examplesof such blended imagery are provided herein. Embodiments describedherein may be configured to uses such blended imagery as the baseimagery for an AI based image processing and object detection andtracking methods. Such methods include providing dewarping andrectification of the two video streams and then runs an object detector(e.g., a CNN based object detector) and tracker on the combined video.The result is a single CNN, which results in better overall efficiencyand compute performance. By contrast, existing systems currently processeach video stream individually, which doubles the compute requirementson the hardware.

Embodiments described herein provide additional performance overconventional techniques because the CNN and tracker can leverageadditional car information (e.g., speed, radar, GPS, direction) as partof the data in the capability of the AI-based processing because theblended video's weighted values of each side can be determined withinthe AI stack. Additionally, the CNN may use additional information fromthe scene temperature data provided by a radiometric thermal camera tocheck average temperatures of classified objects for acceptable range oftemperature to help limit false positive detections. For example, pixelsof an object classified as a person by the CNN all have temperaturevalues associated with the identified person. Average temperature valuesmay be checked against acceptable ranges for the specific type of targetin the associated specific environment.

Night time driving can be difficult for drivers; drivers may find itdifficult to see critical road obstructions and/or other objects likeVulnerable Road Users (VRUs) and animals. Embodiments described hereinprovide additional information to the driver so that the driver can seeVRUs and animals with sufficient warning to react so that an accidentcan be avoided during night and other challenging lighting conditions.For example, blended video with alerts can be presented to the driver,or information from the cameras (e.g., blended, as described herein) anddetection mechanism may be used to stop the vehicle using AEB (automaticemergency brakes).

In various embodiments, critical information from the detection of a VRUcan be displayed to the driver. For example, example display views(e.g., FIGS. 16A-F) highlight pedestrians and vehicles using the thermalcamera, but the visible camera shows the view from the car from thedriver's perspective (e.g., including what the driver's eyes can see).When using a CNN with the blended video, embodiments use informationcombined from thermal and visible spectrum data and reduces computationcosts as compared to two separate processing stacks operatedindependently on the visible spectrum video and the thermal video.

False positive detections are possible with either or both the thermaland visible spectrum imagery. Embodiments described herein use bothvideo channels to more accurately detect VRUs, animals, and otherobjects (like other vehicles) to more reliably determine the presenceand type of object that is classified by the CNN. Embodiments alsodetermine and use additional information corresponding to the averageradiometric thermal values of detected objects to determine if aclassified object is within expected radiometric thermal values toreduce and/or eliminate false positives in the presence and/orclassification of the object.

Certain calibration objects in the scene can also be used as referencetemperatures to help ensure the radiometric thermal values ofnon-calibration objects are valid/calibrated/accurate for othernon-calibration objects (like VRUs) that are detected. For example, acalibration object may include a designated portion of a hood for thevehicle the thermal imaging module is attached to—that designatedportion of the hood may be in the field of view of the thermal camera,and a temperature sensor may be coupled to the designated portion of thehood and provide an accurate temperature of the designated portion ofthe hood, which may be used to calibrate the thermal imaging module.Such calibration object may be marked by IR and/or visible spectrumvisible markers (e.g., registration spots, crosses, and/or othergraphics) to help identify pixels associated with the calibrationobject. In other embodiments, such calibration objects may beimplemented by stationary structures (signs, placards) placed by a roador at an exit of a parking lot or structure that are kept at a standardtemperature or are configured to communicate their temperature to system100, such as through IR or visible spectrum lighted text, for example,or via a wireless beacon.

Various embodiments include a thermal camera (e.g., a thermal imagingmodule) and a visible spectrum camera (e.g., a visible spectrum imagingmodule) with similar or at least overlapping fields of view mounted onor otherwise coupled to a vehicle. The two videos may then be scaled,dewarped, and/or rectified via a calibration process employing aregistration target that both cameras can see. Such calibration processmay be configured to align the pixels between the two camera systems(pixel registration). Visible and thermal lenses/lens systems may havedifferent aspects and the calibration process may compensate for theradial and tangential aspects of each lens so that imagery from the twocameras is substantially aligned and may be superimposed on each otherso that imaged objects are overlapping in each video.

In some embodiments, such registration calibration can be real timeadjusted using objects in the scene with detectable andwell-defined/sharp edges so that alignment can be maintained orcorrected even after initial calibration processes are performed.Aligned imagery may then be displayed to the driver or used by an ADASenabled vehicle, as described herein. Combined imagery (e.g., displayviews and/or combined image data) can include variable contributionsfrom 100% thermal and 0% visible to 100% visible and 0% thermal oranywhere in between, for example. Such variation can be selected basedon ambient lighting, weather, and/or other environmental conditionswhere one spectrum provides more reliable image data with respect to aparticular application (e.g., object detection, classification, imageregistration, and/or other image processing, as described herein).

In various embodiments, registered multi-spectral video (thermal andvisible) can be displayed to a user and/or input into an objectdetection algorithm, specifically a CNN based object detector. Asdescribed herein, blended video provides superior objectdetection/perception as compared to only visible or only thermal video.Embodiments of such CNNs may also implement a tracker that maintains atarget lock between image frames and increases the reliability ofdetection of objects through time. Embodiments of such CNNs arerelatively highly efficient in compute cost due to employing only onenetwork as compared to conventional image processing for each sensortype. The result is a lower cost due to compute hardware and morereliable detection/classifications due to multiple simultaneousspectrums (e.g., thermal and visible).

Radiometric thermal values (calibrated absolute temperatures) per pixelmay also be achieved with thermal imaging modules, as described herein.Such per pixel data may be used when classified objects (e.g., VRU) aredetected, and the average values of the pixels associated with the VRUare combined and relative temperature values are obtained. A thresholdrange may be used to evaluate the object temperatures as compared toexpected values for a particular scene/ambient temperature. Suchprocessing adds confidence to the CNNs capability and helps reduce oreliminate false positives, which is particularly helpful for reliableand comfortable AEB applications.

In various embodiments, display views including blended spectral imagerymay be generated for drivers. In some embodiments, the blended video maybe configured to emphasize street signs and color (from the visiblecamera), and in other embodiments VRUs in thermal are emphasized. Otherdisplay views are primarily visible spectrum imagery with detectedobjects shown in thermal. In other embodiments, the thermal and visibleblended video with CNN detection may be shown to the driver in aconfigurable manner so that a variety of driver scenarios are accountedfor and optimized. For example, in full dark night driving the thermalimage module may provide the majority of the video stream. In welllighted situations, the visible spectrum image module may provide themajority of the transparency. In further embodiments, thermal detectionswill be superimposed on visible spectrum video, especially in conditionswhere oncoming headlights of cars may glare and wash out visiblespectrum imagery.

FIGS. 16A-B illustrate display views 1600-1605 including imagerygenerated by thermal imaging system 1240 for thermal imaging navigationsystem 1200 in accordance with embodiments of the disclosure. Forexample, display view 1600 of FIG. 16A shows blended image 1610 withblended imagery CNN based maneuvering obstacle tags 1620 (oncomingtraffic), 1622 (parked vehicle along lane), and 1630 (pedestrian walkingacross traffic in a cross walk). Each maneuvering obstacle tag indisplay view 1600 includes a labeled box around thermal image datarepresenting the corresponding detected object/maneuvering obstacle. Inthe embodiment shown in FIG. 16A, blended image 1610 is primarilyvisible spectrum imagery combined with the thermal imagery associatedwith maneuvering obstacle tags 1620, 1622, and 1630.

Display view 1601 of FIG. 16B shows blended image 1610 with blendedimagery CNN based maneuvering obstacle tag 1620 (oncoming traffic),which is similar to display view 1600 of FIG. 16A, but presents adifferent environment where the visible spectrum glare of the oncomingheadlights would otherwise obscure the oncoming traffic if that portionof blended image 1610 were not overlaid or otherwise blended with thethermal imagery associated with maneuvering obstacle tag 1620.

Display view 1602 of FIG. 16C shows blended image 1610 with blendedimagery CNN based maneuvering obstacle tag 1632 (oncoming bicyclerider), which is similar to display view 1600 of FIG. 16A, but presentsa different environment where the visible spectrum imagery wouldotherwise not show the oncoming bicycle rider if that portion of blendedimage 1610 were not overlaid or otherwise blended with the thermalimagery associated with maneuvering obstacle tag 1632.

Display view 1603 of FIG. 16D shows blended image 1612 that is primarilythermal imagery combined with visible spectrum imagery associated withvarious illuminated objects, such as taillight 1640, intersection signallight 1642, and roadside reflectors 1644 (e.g., all shown as red inblended image 1612). In the embodiment shown in FIG. 16D, blended image1612 includes detail sufficient to identify and detect pedestrian 1650and lane markings 1660, even in relatively low light conditions (e.g.,where visible spectrum imagery might not include similar details).

Display view 1604 of FIG. 16E shows blended image 1612 that is primarilythermal imagery combined with visible spectrum imagery associated withvarious illuminated objects, such as crosswalk sign 1646, lane diversionsign 1648, corner light 1670 (with associated holiday lighting), andstreetlight 1672 (e.g., all shown in visible spectrum color—white,yellow, orange—in blended image 1612). In the embodiment shown in FIG.16E, blended image 1612 includes detail sufficient to identify anddetect lane markings 1660, even in relatively low light conditions(e.g., where visible spectrum imagery might not include similar detailsor otherwise be overblown by glare of oncoming traffic headlights).

Display view 1605 of FIG. 16F shows blended image 1612 that is primarilythermal imagery combined with visible spectrum imagery associated withvarious illuminated objects, such as taillights/running lights 1640(e.g., shown in visible spectrum color—red—in blended image 1612). Inthe embodiment shown in FIG. 16F, blended image 1612 includes blendedimagery CNN based maneuvering obstacle tags 1624 (relatively small sizedtraffic traveling in the same direction along the road) and 1626(relatively large sized traffic traveling in the same direction alongthe lane). Each maneuvering obstacle tag in display view 1600 includes alabeled box around blended image data representing the correspondingdetected object/maneuvering obstacle along with its associatedilluminated features (e.g., taillights/running lights 1640).

By providing systems and techniques that include CNN based imageprocessing on combined or multispectral imagery, embodiments of thepresent disclosure substantially improve the operational flexibility andreliability of manned and unmanned mobile platforms, including passengervehicles. Moreover, such systems and techniques may be used to increasethe operational safety of users and operators of mobile platforms,including of manned passenger vehicles, beyond that achievable byconventional systems. As such, embodiments provide thermal imagingnavigation systems with significantly increased operational convenienceand performance.

Air path is a significant contributor of radiometric measurement error.If a user desires to know the accurate absolute temperature of an objectat a distance, it is required that they understand the amount ofatmosphere that the infrared radiation is traveling through, the makeupof that atmosphere (water content, etc.), and air temperature. In anADAS application, multiple sensors (LIDAR, Radar, stereovision) can beused to determine the distance to an object of interest. If the distanceis known, an accurate estimation can be made of the signal lost in theair path between the camera and the object of interest. This cansignificantly improve radiometric accuracy.

LIDAR and RADAR are time of flight ranging sensor systems that can beused to determine depth. Stereovision camera systems can determine depthby triangulating objects common in two fields of view with a knowndistance between image planes. Temperature and humidity sensors in theopen air can be used to estimate water content and temperature of theair path. In various embodiments, temperature and humidity data can beused to estimate the signal coming from the air path per unit length.One of the ranging sensor systems above can be used to determine thedistance between the IR camera and an imaged object of interest. Thisdistance can be multiplied against the air path per unit length todetermine the amount of signal coming from the air path.

It is important to understand what the required range is for anautomotive application to understand why infrared camera radiometricaccuracy at range is required. US, Germany, and UK stopping distancesare compared and a vehicle needs ˜160 meters to stop if traveling at 70MPH. In the automotive environment, most concerning are targets that areup to ˜160 meters away assuming the vehicle is traveling at a top speedof ˜70 MPH. Therefore, embodiments should accurately classify people outto ˜160 meters. Identification of a human target can be done usingcontrast of target relative to background, using shape identification ofthe target—however, embodiments within are more focused on using theabsolute temperature of the target using a radiometrically calibratedcamera as a means to determine whether a target is human.

Measuring the absolute temperature of a target at range using aninfrared camera is complicated by the air path. The atmosphere operatesas a filter that absorbs energy emitted by the target, and theatmosphere itself also emits light that looks like signal from theperspective of the infrared camera. As shown in FIG. 17A, the errorcomponents in the radiometric image due to reflectance of the atmosphereand atmospheric transmission losses may be determined analytically, asshown by diagram 1700. Many models exist to model atmospherictransmission effects, and the impacts of humidity and ambient airtemperature on atmospheric transmission can be determined, as shown inplot 1702 of FIG. 17B. When considering ranges of air ambienttemperature between 20° C. and 40° C., with relative humidity between 5%and 100%, the energy that reaches a thermal sensor is somewhere between0.93 (93%) and 0.53 (53%) with a target out to 500 meters. Therefore,when considering stopping distances of ˜160 meters, air transmission canbe between 0.95 (95%) and 0.75 (75%).

Considering 37° C. target (human body temperature w/˜emissivity of 1) inhigh humidity/high ambient conditions, a human can roughly be observedas 36.63° C. at one meter range to as low as 27.75° C. at 160 meters inthe same ambient and relative humidity conditions—27.75° C. is wellbelow a normal human body temperature and not usable for humandetection. As seen in the atmospheric transmission plot 1704 of FIG.17C, H2O in the atmosphere can absorb a significant amount of energy,and it is variable with atmospheric conditions, which is why it isimportant to have an accurate estimate of humidity for radiometricaccuracy. Water has an absorption band in the 8-13.5 micron band thatcan contribute to signal loss in the air path. Relative humidity can beused to estimate how much that water that is present in the atmosphereis blocking signal from the target down range.

Embodiments disclosed herein use on-board vehicle sensors (e.g., intakeair temperature sensor 1810 and humidity sensor 1812 of FIG. 18, alongwith corresponding measurement data) for atmospheric humidity andtemperature estimation for the purposes of correcting radiometricmeasurement errors in an infrared camera due to air path losses whenmeasuring absolute temperature of a target of interest (e.g., apedestrian) and/or detecting and classifying such target at long range(˜160 meters) for the purposes of stopping the vehicle at high speed ortaking evasive action. Off-the-shelf sensors are typically integratedwith vehicles already and are configured to keep track of ambientrelative humidity and air temperature. Air and humidity as measured bythe ECU for things such as fuel mixture optimization can be used for thealternate purpose for a radiometric flux to temperature conversionoccurring on the IR camera.

In addition, GPS and/or a clock can also be used with an almanac—apriori knowledge about typical weather patterns in local climates—for arough estimate of expected air humidity and air temperature. In additionto on-vehicle sensors, most vehicles employing modern sensors are alsoconnected to the internet. Hyper-local temperature and humidity data isavailable to the vehicle in most locations that support internetconnections.

One method of target depth measurement for the purpose of estimating airpath losses for improving radiometric accuracy is LiDAR. LiDAR iscomposed of a laser source and a detector—the “time of flight” for alaser pulse is measured and from that measurement in time, distance totarget can be inferred. In a sensor-fused system, the IR camera andLiDAR have overlapping fields of view—every pixel covered by IR willhave a distance measured via LiDAR. Because theoretically every IR pixelin the scene has a corresponding distance to the target emitting IRenergy, atmospheric transmission can be calculated on a per pixel basisfor the entire scene, allowing for accurate temperature measurement forall objects in scene, including objects that are over 100 meters away inhigh humidity/high temperature environments.

Another method of target depth measurement for the purpose of estimatingair path losses for improving radiometric accuracy is RADAR. RADAR iscomposed of a radiowave source and a detector—RADAR is a “time offlight” sensor superset which includes LiDAR. W band imaging RADAR iscommon in autonomous vehicle development. In a sensor-fused system, theIR camera and RADAR will have overlapping fields of view—every pixelcovered by IR will have a distance measured via RADAR. Becausetheoretically every IR pixel in the scene has a corresponding distanceto the target emitting IR energy, atmospheric transmission can becalculated on a per pixel basis for the entire scene, allowing foraccurate temperature measurement for all objects in scene, includingobjects that are over 100 meters away in high humidity/high temperatureenvironments.

Another method of target depth measurement for the purpose of estimatingair path losses for improving radiometric accuracy is the use a stereocamera array. Using well known methods and two cameras in a fixedposition relative to each other, with image planes that have beencalibrated such that they are known to be parallel to each other andaligned in Z with known focal lengths, the distance to target can beestimated using a calculation based on equivalent triangles. In astereo-camera based depth measurement system, two IR cameras, or twovisible cameras, will have overlapping fields of view—every pixelcovered by IR will have a distance measured via stereo-vision. As above,because theoretically every IR pixel in the scene has a correspondingdistance to the target emitting IR energy, atmospheric transmission canbe calculated on a per pixel basis anywhere that there is scenecontrast, allowing for accurate temperature measurement for most objectsin scene, including objects that are over 100 meters away in highhumidity/high temperature environments.

Display view 1900 of FIG. 19 includes left and right images 1910 and1912 of a stereo thermal camera system, along with a rough (noisy)disparity map 1920, where closer objects found in left and right imagesare shown as brighter and objects that are farther away have darkerpoints on the disparity map. Such disparity map may be used to thendetermine a range to various objects (e.g., an average range if averagedacross all pixels of a detected/classified object) in an imaged scene.

Embodiments described herein may be configured to create a thermal depthmap using a single thermal camera and the reflection off of a carhood—using reflectivity in LWIR off a car hood as a source for a secondimage plane that can be used as a virtual camera.

In automotive ADAS applications like AEB, it is important to understandthe distance of objects in front of the vehicle. LWIR (e.g., thermal IR)is useful for identifying objects in challenging whether conditions.Disclosed herein are techniques of determining distance to objectsidentified in thermal imagery using a single thermal imaging modulesensor by taking advantage of the reflectivity of the car's hood. Inroof-mount applications, such as that presented in system 2000 of FIG.20, the car hood is in view of the thermal camera along with objects tobe identified in front of the vehicle—by determining distance using asingle thermal camera and reflective car hood, overall system cost canbe reduced (no need to install two thermal cameras, or thermal+RADAR, orthermal+Visible).

In roof mounted thermal imaging applications, the hood of the vehiclecan be within the field of view that includes the general navigationscene. Depending on its emissivity, the hood itself can present areflection of the scene in front of the vehicle back at the thermalcamera. Because the hood and thermal camera are fixed in space, theimage reflected from the hood can be treated like a “virtual camera” andthe distance between the “virtual camera” and the real camera can beused to calculate a disparity map between objects common to both images.For example, the hood image may be used to triangulate distance just asin a stereo imaging system. The hood is a known fixed distance from thethermal camera and it moves coincident with the thermal camera, so thereflection in the hood provides information as if it's a secondarycamera in a stereovision system—this allows one to create a depth mapfrom reflections off of vehicle hood.

In system 2000 of FIG. 20, a roof mounted thermal camera is shown with avertical field of view. A portion of the field of view of the roofmounted camera intersects the vehicle hood. In data collected fromthermal cameras mounted in such a manner, the hood of the vehicle canact as a mirror in the LWIR spectrum. This mirror has the effect ofcreating a second image plane at a different point in space, which canbe viewed as a “virtual” thermal camera. Because the reflective vehiclehood and roof mounted thermal camera are fixed in space relative to eachother, the “virtual” thermal camera/secondary image plane can also betreated as fixed and the real and virtual thermal cameras can be treatedas a stereo pair for the purposes of depth estimation using a disparitymap or other such method in the areas of the real world scene where thereal thermal camera and virtual thermal camera have intersecting fieldsof view.

There is a structural dependence on curvature of the hood, detectabilityrelated to emissivity/reflectivity of the hood, and additionalstructural dependence upon the mounting height relative to the hood andpose of the thermal camera on the roof. A major benefit of this approachis that a single camera can be treated as two cameras with simplercalibration than a traditional stereo pair and no synchronizationrequirement as both views of the same scene are captured using a singlesensor.

In diagram 2100 of FIG. 21, calculations are provided that show how toestimate depth in a stereo pair system. An object common to both viewswill have a vertical shift between both real and virtual image planesthat is dependent upon the distance between both cameras from eachother. In the example illustration, the mirror axis between the realthermal camera and the virtual thermal camera is about halfway betweenthe roof and hood of the vehicle due to the hood angle. It can then bedetermined that the distance between the real image plane and virtualimage plane is about equal to the distance between the roof and hood, or˜1 meter. In various embodiments, distance this would be preciselymeasured and known. In the case where a wide-angle lens is used on thethermal camera, the focal length could be approximately 6.2 mm—because amirror is used, the virtual thermal camera can be assumed to have thesame focal length of 6.2 mm.

To calculate disparity, a feature of an object of interest (like apedestrian limb or head) needs to be identified in both image planesusing CNN or some other image processing technique, as described herein.Once this has been done in the real and virtual image plane, thedifference between centroids of the features identified as common in thevertical orientation can be saved as the disparity of the feature asseen from two points of view. This disparity, along with known focallength and distance between real and virtual cameras can be used toestimate depth by treating the problem as one of equivalent triangles.In the example shown in FIG. 21, an object at 160 meters as imaged by amirror system composed of a hood reflection and a 12 um camera with afocal length of 6.2 mm and image plane difference of 1 mm will see avertical disparity of 3.23 pixels between both image planes. An objectat 50 meters would look like a vertical disparity of 10.3 pixels betweenboth image planes.

FIG. 22 illustrates a flow diagram 2200 of various operations togenerate a range map using a thermal imaging navigation system inaccordance with embodiments of the disclosure. More specifically, flowdiagram 2200 illustrates the process to rectify and estimate targetdepth. The mirrored image of the scene cannot be used as is and willrequire at least one flip along the mirror axis such that it is orientedin the same orientation as the real image plane. Due to the hood contourand roll off towards the end of the hood, some warping will occur. Thiswarping can be removed and should only need to be calibrated once as thehood shape and pose of the camera relative to the hood is not expectedto change over time.

After the image is rectified, common features between the real imageplane and virtual image plane will be found. The real image plane imagecan be cropped such that only the stereo region is analyzed before thefeature detection step. One approach could be to run a CNN against bothreal and virtual images—the centroid of the bounding box drawn ondetected objects can potentially be used for the disparity calculation.Another method is to use a Harris Corner Detector. After common featuresare mapped, their difference in the vertical orientation can becalculated and depth can be estimated using the equation previouslydetailed.

FIGS. 23A-C illustrate display views including imagery generated by athermal imaging system for a thermal imaging navigation system inaccordance with embodiments of the disclosure. Display view 2300 of FIG.23A shows a pedestrian in a hallway, the stereo region of the realimage, and the rectified mirror image (no attempt was made here to fixwarping due to hood contour). Display view 2302 of FIG. 23B shows a roadsign on a turn, the stereo region of the real image, and the rectifiedmirror image (no attempt was made here to fix warping due to hoodcontour). Display view 2304 of FIG. 23C shows car headlights on theopposite side of the road, the stereo region of the real image, and therectified mirror image (no attempt was made here to fix warping due tohood contour). Each display view may include blended thermal and visiblespectrum imagery, for example, such as the visible spectrum colorilluminated objects (road lights) shown in FIG. 23C.

In some embodiments, the hood of the vehicle may include a radiometriccalibration object, such as a portion of the hood that is marked and/orcoupled to a thermal sensor so that the temperature of the portion ofthe hood may be provided to system 2000/100 and the correspondingthermal camera calibrated, as described herein.

Where applicable, various embodiments provided by the present disclosurecan be implemented using hardware, software, or combinations of hardwareand software. Also, where applicable, the various hardware componentsand/or software components set forth herein can be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein can be separated into sub-components comprising software,hardware, or both without departing from the spirit of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components can be implemented as hardware components, andvice-versa.

Software in accordance with the present disclosure, such asnon-transitory instructions, program code, and/or data, can be stored onone or more non-transitory machine-readable mediums. It is alsocontemplated that software identified herein can be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein can be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

Embodiments described above illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the present invention.Accordingly, the scope of the invention is defined only by the followingclaims.

1. A system comprising: a thermal imaging device configured to bemounted on a vehicle, wherein the thermal imaging device is configuredto, when mounted on the vehicle, capture a first image of a sceneencompassing a portion of the vehicle and capture a second imageassociated with a reflection of the scene from the portion of thevehicle; and a logic device configured to communicate with the thermalimaging device and determine a disparity map based on the first imageand the second image.
 2. The system of claim 1, wherein the logic deviceis further configured to: identify at least one object common to thefirst image and the second image; and determine a disparity associatedwith the at least one object based on a shift between the at least oneobject in a first image plane associated with the first image and the atleast one object in a second image plane associated with the secondimage, wherein the disparity map comprises the disparity associated withthe at least one object.
 3. The system of claim 2, wherein the logicdevice is further configured to determine a depth associated with the atleast one object based on the disparity associated with the at least oneobject and a distance between the thermal imaging device and a virtualimaging device associated with the second image plane.
 4. The system ofclaim 1, wherein: the thermal imaging device and the portion of thevehicle are fixed in position relative to each other, the logic deviceis further configured to flip the second image along an axis to obtain amirrored image, and the logic device is configured to determine thedisparity map based on the first image and the mirrored image.
 5. Thesystem of claim 1, wherein: the thermal imaging device is configured tobe mounted on a roof of the vehicle, the portion of the vehiclecomprises a portion of a hood of the vehicle, the logic device isfurther configured to de-warp the second image based on a shape of thehood and a pose of the thermal imaging device to obtain a de-warpedimage, and the logic device is configured to determine the disparity mapbased on the image and the de-warped image.
 6. The system of claim 1,wherein the logic device is further configured to determine path lossesbased on the disparity map and determine radiometric data associatedwith the scene based on the disparity map.
 7. The system of claim 1,wherein: the thermal imaging device is further configured to providethermal image data corresponding to a projected course for the vehicle;and the logic device is further configured to receive the thermal imagedata corresponding to the projected course.
 8. The system of claim 7,further comprising: a sensor system coupled to the vehicle andconfigured to provide sensor data associated with the projected course;wherein the logic device is configured to receive the sensor datacorresponding to the thermal image data, wherein the projected course isbased, at least in part, on a combination of the sensor data and thethermal image data.
 9. The system of claim 8, further comprising acommunication device configured to establish a wireless communicationlink with an update server associated with the vehicle, wherein: thelogic device is further configured to receive the thermal image datafrom the thermal imaging device as the vehicle maneuvers along theprojected course and report information corresponding to the projectedcourse over the wireless communication link to the update server. 10.The system of claim 8, wherein the sensor system comprises anorientation sensor, a position sensor, a visible spectrum imagingsystem, and/or a ranging sensor system, and wherein the sensor datacomprises visible spectrum image data corresponding to the projectedcourse, orientation data associated with motion of the vehicle, positiondata associated with motion of the vehicle, and/or ranging sensor datacorresponding to the projected course.
 11. The system of claim 10,wherein: the sensor data comprises the visible spectrum image data andthe logic device is further configured to generate blended imagerybased, at least in part, on the visible spectrum image data and thethermal image data; and/or the sensor data comprises the ranging sensordata and the ranging sensor system comprises a grille mounted radarsystem and/or a grille mounted lidar system.
 12. A method comprising:capturing, by a thermal imaging device mounted on a vehicle, a firstimage of a scene encompassing a portion of the vehicle; capturing, bythe thermal imaging device, a second image associated with a reflectionof the scene from the portion of the vehicle; and determining adisparity map based on the first image and the second image.
 13. Themethod of claim 12, further comprising: identifying at least one objectcommon to the first image and the second image; and determining adisparity associated with the at least one object based on a shiftbetween the at least one object in a first image plane associated withthe first image and the at least one object in a second image planeassociated with the second image, wherein the disparity map comprisesthe disparity associated with the at least one object.
 14. The method ofclaim 13, further comprising determining a depth associated with the atleast one object based on the disparity associated with the at least oneobject and a distance between the thermal imaging device and a virtualimaging device associated with the second image plane.
 15. The method ofclaim 12, further comprising flipping the second image along an axis toobtain a mirrored image, wherein the disparity map is based on the firstimage and the mirrored image, wherein the thermal imaging device and theportion of the vehicle are fixed in position relative to each other,wherein the thermal imaging device is mounted on a roof of the vehicle,and wherein the portion of the vehicle comprises a portion of a hood ofthe vehicle,
 16. The method of claim 12, further comprising: determiningpath losses based on the disparity map; and determining radiometric dataassociated with the scene based on the disparity map.
 17. The method ofclaim 12, further comprising capturing, by the thermal imaging device,thermal image data corresponding to a projected course for the vehicle.18. The method of claim 17, further comprising receiving sensor datacorresponding to the thermal image data, wherein the projected course isbased, at least in part, on a combination of the sensor data and thethermal image data.
 19. The method of claim 18, wherein the sensor datacomprises visible spectrum image data corresponding to the projectedcourse, orientation data associated with motion of the vehicle, positiondata associated with motion of the vehicle, and/or ranging sensor datacorresponding to the projected course.
 20. The method of claim 17,wherein the thermal image data is captured as the vehicle maneuversalong the projected course, wherein the method further comprises:reporting information corresponding to the projected course over awireless communication link to an update server associated with thevehicle via a communication device configured to establish the wirelesscommunication link with the update server.